Does Java 8 Still Need LINQ? Or is it Better than LINQ?


Photo by Ade Oshineye

Erik Meijer, Tye Dye Expert. Photo by Ade Oshineye. Licensed under CC-BY-SA

LINQ was one of the best things that happened to the .NET software engineering ecosystem in a long time. With its introduction of lambda expressions and monads in Visual Studio 2008, it had catapulted the C# language way ahead of Java, which was at version 6 at the time, still discussing the pros and cons of generic type erasure. This achievement was mostly due to and accredited to Erik Meijer, a Dutch computer scientist and tye-dye expert who is now off to entirely other projects.

Where is Java now?

With the imminent release of Java 8 and JSR-355, do we still need LINQ? Many attempts of bringing LINQ goodness to Java have been made since the middle of the last decade. At the time, Quaere and Lambdaj seemed to be a promising implementation on the library level (not the language level). In fact, a huge amount of popular Stack Overflow questions hints at how many Java folks were (and still are!) actually looking for something equivalent:

Interestingly, “LINQ” has even made it into EL 3.0!

But do we really need LINQ?

LINQ has one major flaw, which is advertised as a feature, but in our opinion, will inevitably lead to the “next big impedance mismatch”. LINQ is inspired by SQL and this is not at all a good thing. LINQ is most popular for LINQ-to-Objects, which is a fine way of querying collections in .NET. The success of Haskell or Scala, however, has shown that the true functional nature of “collection querying” tends to employ entirely other terms than SELECT, WHERE, GROUP BY, or HAVING.  They use terms like “fold”, “map”, “flatMap”, “reduce”, and many many more. LINQ, on the other hand, employs a mixture of GROUP BY and terms like “skip”, “take” (instead of OFFSET and FETCH).

In fact, nothing could be further from the functional truth than a good old SQL partitioned outer join, grouping set, or framed window function. These constructs are mere declarations of what a SQL developer would like to see as a result. They’re not self-contained functions, which actually contain the logic to be executed in any given context. Moreover, window functions can be used only in SELECT and ORDER BY clauses, which is obvious when thinking in a declarative way, but which is also very weird if you don’t have the SQL context. Specifically, a window function in a SELECT clause influences the whole execution plan and the way indexes are employed to pre-fetch the right data.

Conversely, true functional programming can do so much more to in-memory collections than SQL ever can. Using a SQLesque API for collection querying was a cunning decision to trick “traditional” folks into adopting functional programming. But the hopes that collection and SQL table querying could be interfused were disappointed, as such constructs will not produce the wanted SQL execution plans.

Conversely, true functional programming can do so much more to in-memory collections than SQL ever can. Using a SQLesque API for collection querying was just the wrong decision. The hopes that collection and SQL table querying could be interfused were disappointed, as such constructs will inevitably produce horrible SQL execution plans.

But what if I am doing SQL?

It’s simple. When you do SQL, you have two essential choices.

  • Do it “top-down”, putting most focus on your Java domain model. In that case, use Hibernate / JPA for querying and transform Hibernate results using the Java 8 Streams API.
  • Do it “bottom-up”, putting most focus on your SQL / relational domain model. In that case, use JDBC or jOOQ and again, transform your results using the Java 8 Streams API.

This is illustrated more in detail here: http://www.hibernate-alternative.com

Don’t look back. Embrace the future!

While .NET was “ahead” of Java for a while, this was not due to LINQ itself. This was mainly due to the introduction of lambda expressions and the impact lambdas had on *ALL* APIs. LINQ is just one example of how such APIs could be constructed, although LINQ got most of the credit.

But I’m much more excited about Java 8’s new Streams API, and how it will embrace some functional programming in the Java ecosystem. A very good blog post by Informatech illustrates, how common LINQ expressions translate to Java 8 Streams API expressions.

So, don’t look back. Stop envying .NET developers. With Java 8, we will not need LINQ or any API that tries to imitate LINQ on the grounds of “unified querying”, which is a better sounding name for what is really the “query target impedance mismatch”. We need true SQL for relational database querying, and we need the Java 8 Streams API for functional transformations of in-memory collections. That’s it. Go Java 8!

Tags: , , , , , , , , , , , , , , ,

20 responses to “Does Java 8 Still Need LINQ? Or is it Better than LINQ?”

  1. agentgt says :

    Lukas this is a great posting/article!

    I think Java 8 brings great promise with its streaming transformations and lambda. I firmly believe that what makes Java rather decent is that it is not really a good OOP language (OOP practices like inheritance are eschewed) but an evolving functional component language with far easier to digest typing then say Scala. More and more are using Java surprisingly as a system domain transformation language. Whether its JAX-RS, Spring Batch, or even old JMS: Java is good at transformation, components (aka services), and immutable objects which are ideal for large scaling transformation problems (ala Map Reduce etc…).

    While LINQ and SQL are very cool they are not really about transformation but querying. I think the heart of FP though is transformation (aka mapping and reducing) and thus fits most of the very challenging cloud/enterprise level problems that startups to large corporations are faced with.

    • lukaseder says :

      Hi Adam,

      Thanks for your nice words. I like your point about FP being mainly about transformation instead of declarative querying. I should’ve better stressed that in the article.

      And please, help me on this reddit bashing that is going on, currently :-)
      http://www.reddit.com/r/programming/comments/1pqscn/does_java_8_still_need_linq_or_is_it_better_than/. I just think that every C# Yes Man out there who has never challenged Erik Meijer’s work is about to beat me good time ;-)

      • Uffe Seerup says :

        Your assumption (on which you build your entire article) about the relationship between LINQ and SQL is fundamentally wrong, hence your article comes across as ill-informed. That is why the article is being bashed, AFAICT.

        • lukaseder says :

          The article has been bashed for many reasons, but mostly by the FP / LINQ corner on reddit. It has had rather positive feedback in the Java community. In my opinion, my assumptions aren’t so wrong. Erik Meijer made a compromise when he designed LINQ. He wanted to be able to query RDBMS, XML, and objects. LINQ is an OK compromise for these query targets, but I really don’t think that a unified querying language and API is what is needed in the long run, as these data stores (and the possible “query” feature scope) are heterogeneous. Which is why I’m calling out for this to be a new impedance mismatch, after the ORM impedance mismatch.

          This doesn’t mean that LINQ is bad. It’s very good. But inappropriate for SQL as it heavily reduces its feature set and expressiveness for those who need SQL. Those who don’t “need” SQL are fine with LINQ-to-SQL as it allows for a faster time-to-market.

          But anyway, I’m repeating myself… ;-)

          • Uffe Seerup says :

            Piece of advice: The fact that you refer to LINQ-to-SQL is a giveaway that you are not really up to date on LINQ. LINQ-to-SQL (LINQ-to-SQLServer would be a more appropriate name) was implemented by the LINQ team because they needed to showcase LINQs potential against a RDBMS. The current state of art in LINQ to databases is LINQ to Entities, a full-blown ORM leveraging LINQ to query databases. Look to that for what LINQ is inherently capable of.

            Then a few points:

            1) LINQ is not a “compromise”. It was designed as a generalized query facility. The team under Erik Meijer researched and isolated a number of standard query operators from a theoretical perspective. *You* claim that they wanted to query RDBMS, XML and objects. IMO that is belittling Erik Meijer and his team’s work. They set out to crack the problem of querying in general, believing that once they had that in the bag, they could create a number of concrete realizations against present and future targets. The standard query operators are the operators through which all other query functions/operators can theoretically be built. Even aggregate or windows functions.

            This also means that LINQ is not a relational query model. It is just a query model. A testament to the strength and generality of the query model is seen in how it has been adopted elsewhere, like in Java’s EL. Another indication of the strength of the model is how easily it could be adapted to the domain of reactive push collections which led to Reactive Extensions.

            2) The fact that someone had to explain to you how expression trees work is disconcerting. It raises the suspicion that you have not fully grasped what LINQ really is. This is important because especially on the topic of mapping queries to RDBMSs, understanding expression trees (ASTs) is crucial. With expression trees a provider for a query target can discover patterns and map to the target. The query provider can provide its own functions and this is how LINQ to Entities implement aggregate functions, and how it *could* be used to map window functions as well. Note, a query provider is a framework implementation. Entity Framework didn’t exist when LINQ was originally released. The current state of EF6 has been implemented using the extensibility points of the original LINQ model. As an example on how the mapping to SQL is extensible through simple framework features, consider the methods of the EF6 DbFunctions. Note how those functions include aggregate functions. These functions have all been defined as library features.

            3) You will always be able to point to some SQL specific feature (or even Oracle specific feature) that cannot currently be mapped (or should not be mapped) to a generalized query or language library, whether that is LINQ, JOOQ, Hibernate or whatever. However, the question is whether the subset of features surfaced to a programmer through LINQ or similar generalized query model is sufficient and whether those features can be reached through some extensibility or escape mechanism.

            4) Whether you believe that a unified query model is what is needed in the long run is a matter of opinion. You are of course entitled to your opinion that we should write SQL instead of Java or C#. I can tell you that scores of .NET developers reap the benefits of LINQ (to Entities/to SQL/to NHibernate/to Objects) every single day. My opinion is that LINQ is what SQL should have looked like. I would prefer that RDBMSs nateively understood some representation of LINQ instead of SQL. By this I do not mean that RDBMSs should understand C# or accept .NET; I merely refer to the standard query operators and the inherent composability.

  2. Aurélien says :

    http://www.hibernate-alternative.com haha that’s clever ;)

  3. Sergio says :

    Lukas, do you know LINQ is functional? LINQ is a Monad, functional programming is not about map or reduce. Functional programming is about composition, you know functions do compose. Monads are about composition and generalisation of concepts at a higher level of abstraction. By the way map is a special case of reduce. Reduce and co. are general purpose algorithms to process collections. LINQ can be extended to include mappings and reductions. Erik Meijer put the familiar words of select, group-by so people won’t be scared by the FP jargon, like Catamorphism. Simon Payton Jones, one of the Haskell creators, wants to add many stuff from LINQ into Haskell’s list comprehensions AKA ZF-Expressions. Don’t let your self confused by words. LINQ is about composition and collections, it isn’t about SQL.

    • lukaseder says :

      Hi Sergio,

      Thanks for your feedback. Yes I’ve got my bashing already on reddit for not having stressed often enough that I’m mostly criticising using LINQ for stuff like LINQ-to-SQL. It doesn’t help stressing the fact that I find LINQ’s monadic comprehensions very good at the beginning of the article. Readers seem to just skip some parts and get caught up in my critique about the actual terms used. I agree, that Erik Meijer was quite cunning in re-using SQL terms to get greater adoption. He says so himself, here.

      Nonetheless, as a SQL guy, I find the LINQ “abstraction” a rather poor fit for actual SQL, and the critique is about SQL / LINQ-to-SQL. After all, according to the above citing of Erik Meijer, SQL is a main target for LINQ, and has always been in its early design phase:

      Programmers in the real world wrestle everyday to overcome the impedance mismatch between relational data, objects, and XML.

      In particular, my claim is that the Java 8 Streams API is a better general fit, because it doesn’t pretend to be querying / transforming anything other than collections (which is the good part of LINQ), and leaves the relational data querying to the libraries suited for that task.

      Obviously, trying to make people think about “accepted facts” is disruptive and leads to people thinking that the writer is “uninformed”… I can live with that. :-)

      Cheers
      Lukas

    • agentgt says :

      Sergio, Advance apologies on calling you out but I disagree with almost entirely what your saying and in general shocked how such an innocuous blog posting has angered so many recent indoctrinated C#-to-FP lovers.

      Functional Programming is not about composition but transformations and monads are actually rather orthogonal to traditional functional programming. Monads are about evaluation order and category theory. They fit nicely with FP but you do not need Monads for FP.

      What Lukas is criticizing about LINQ is it looking like SQL. This is known in academics as “Cognitive Dissonance”. That it is confusing because you have some ideas how something should work because of previous experiences with something different but looks very very similar.

      I’ll give you an example that happened in the Java community of cognitive dissonance: GWT. GWT looks like Java but its not Java. It compiles down to Javascript and even very efficient Javascript. Some people liked GWT but I would say most found it confusing and somewhat misleading as its not really Java and the mapping was and is not 1-to-1. Also most felt the pure Javascript approach led to greater flexibility.

      Java does actually have a generic language to query objects. Its called JQL (aka JPA query language). It looks like SQL but its not SQL. And yeah its not compile time safe but thats because Java doesn’t let you do AST extending (you can in Groovy extend the AST and thus make Monads and Scala also has something similar). But it turns out many people hate JQL (also known has HQL) because its not really SQL and hence one of the reasons jOOQ is so popular.

      The Java FP enhancements are embarrassingly simple and yet much needed addition to the language. While evaluation order abstractions and language AST extending are powerful all too often it can easily become DSL hell. Everyone starts implementing crappy domain specific language for working in the domain of… another language … why not just use the other language ;) (luckily jOOQ is not crappy and is very 1-to-1) . Also as Scala and Clojure have shown you don’t need an extension to the language to have a better/easier/concise collection processing: you just need a better collection library that is FP friendly. Thats hopeful what will slowly happen to Java.

      That being said LINQ is an amazing piece of tech and most in the Java community are envious but I’m not sure I want Oracle coming up w/ a LINQ alternative.

  4. Adam says :

    I think there is a real misunderstanding of what LINQ is.

    1.) It isn’t based on SQL, it’s based on FLOWR.
    2.) You can write very complex SQL queries with a few lines of LINQ.
    3.) You can do memory queries, which can be very powerful when querying subsets of data from other sources.
    4.) Ironically, it is the point in time when C# starts to get a little bit more functional.

    I’m a C# and Java programmer; but I’m not a champion of either, or anything really. Infact I didn’t really like the concept of LINQ to begin with, I just thought it was syntactical sugar.

    Many years on and I can do things quicker and tidier in C#.

    Microsoft doesn’t get everything right and I think there is an argument that C# started as a Java clone (I got into it by just substituting some of the keywords.) Time to steal some ideas back.

    • lukaseder says :

      It’s based on both SQL and FLWOR (the latter itself being based on SQL). Agreed, there are for, in, and let keywords in LINQ, which were borrowed from FLWOR. But there is also GroupBy, Cross Join, Left Outer Join, Distinct, which are hardly part of FLWOR (as in For, Let, Where, OrderBy, Return.

      The reason (I think) for blending both languages is again to be seen in Erik Meijer’s main idea behind LINQ:

      Programmers in the real world wrestle everyday to overcome the impedance mismatch between relational data, objects, and XML.

      So, if a universal querying API / language should be able to standardise queries against relational data AND XML, you better blend SQL with XQuery to keep the added impedance mismatch at a minimum.

      This isn’t just ironically the point in time when C# started to get a little bit more functional. LINQ was a cunning way to introduce functional programming to “mainstream” developers (again citing from Erik Meijer himself).

      Note that this article absolutely doesn’t challenge the benefits LINQ has brought to .NET in general. I’m looking forward to Java 8’s catching up with all the other platforms. But it challenges LINQ’s appropriateness for SQL from a very SQL-centric developer’s point of view. Yes, it’s easier to navigate paths in LINQ, which produce chained LEFT OUTER JOINs and semi-joins / anti-joins using [NOT] EXISTS. But SQL itself has become much much better and more expressive, which is not reflected in LINQ-to-SQL.

      • Adam says :

        LINQ-to-SQL isn’t to replace SQL with LINQ. It’s an ORM. That’s why you have Stored Procedure support and that’s also why you can execute SQL queries in a LINQ-to-SQL data-context.

        I think LINQ-to-SQL comes into it’s own when you use it in an MVC or MV-anything pattern. That’s why you see similar (but not identical) functionality ‘out-of-the-box’ in node.js and Ruby-on-Rails.

        I wouldn’t be as harsh as to say “Java needs to wake up and smell the coffee” but something like LINQ would be useful in Java as long as you can get back to SQL when you need it.

  5. Tec Goblin says :

    Some things need to be clarified: we all understand that LINQ is a catch all name of a lot of features of C#. Many of those features provide very strong functional concepts, some of them having no correspondence even in Java 8 (Of course Java 8 has some functional concepts C# doesn’t have, but still in my every day programming, I prefer the limited type inference of C# over Java’s functional interfaces).
    That said, the point this article wanted to make seems to be that Java 8 doesn’t need LINQ, ie some of the features LINQ has and Java 8 doesn’t, mostly the built-in language shortcuts that map to SQL-like notions. I disagree.
    The point is not whether Java 8 CAN do enough functional programming. The point is whether the ecosystem (developers, projects, frameworks) will. The ecosystem IS late. Most Java projects are enterprise dinosaurs, with developers used to doing things the way they always did (I am in a Struts environment, for example). In that sense, Java needs all the help it can get to actually get the engine going and motivate all those people to enrich their way of thinking and developing.
    Java Streams are not enough, because they’re not the trojan horse LINQ is. LINQ was a nice, expressive and type-safe abstraction over SQL (and, wow, it had autocomplete, even in free versions of Visual Studio). For most cases it was an imprpovement over SQL, that could get addicting. Plus, it got whole-hearted support by Microsoft (LINQ to Entities/Hadoop/Observables/whatever you want), of a kind the distributed Java ecosystem cannot provide. It took around 5 years to become a staple of almost all development teams. At that point, somebody searches a bit more, or comes with a functional programming background and tells the team “Hey, we’re almost doing functional programming! What if we started creating our own functions?”.
    The only similar trojan horse Java has (and it’s not an exclusivity in any way) is JavaScript. A lot of people get used to passing functions around by using jQuery. But people doing back end? How do you get them on board?
    With Java 8 coming so late and incomplete, I fear that most java projects will become cobol-like snails. Only a few teams with key people with strong academic or recent .NET /ruby or objective-c background will be able to use a good part of the new possibilities, and even there it’s often an uphill battle of a lone developer trying to change things.

    • agentgt says :

      Tec,

      So what your saying is because Java 8 doesn’t have LINQ its going to become Cobol and only a few will know it. Seriously! Java is not going to die because it doesn’t have LINQ.

      While Java the language is behind, infrastructure wise it is not. I think Microsoft .NET has a seriously problem in that most cloud environments you cannot use it in. Also a large portion of the .NET world is not doing SaaS and still doing shrinkwrap-ware or in-house (ie product installs and thus ie can see how manager would prefer LINQ or any ORM for database agnostic access).

      > How do you get them on board?

      A long time ago when Java was adding Generics, people in the C++ house said that no one would use Java’s weak generics and that Java people are too stupid to use Generics and it was way too late to add them. Today I don’t know of library that doesn’t use generics. I expect in 8 or so years most Java libraries if not all will be using Java 8’s features. Yeah 8 years is a long time but Java is a massive community with a massive code base. My bet is in that time frame Java, C#, C and Javascript will still be the top four languages.

      One valid point I think your trying to make with the Javascript reference (otherwise I have no idea what your point is) is that Java might turn into callback hell for concurrent processing. Yes this is a problem with many different platforms (including Node.js) and .NET does have a very cool way of doing it with Rx Observables. That being said the Java way is to do it with a library (see RxJava) . But IIRC to go back to Lucas and my point of cognitive dissonance, I think the Rx Observables don’t even fully support LINQ.

      Finally the Java community sort of believes in the library approach partly because of being burned by Oracle and partly because libraries are much safer, require less resources and more backwards compatible than extending the language or adding some unified approach to data access. Java is thus somewhat analogous to the Unix philosophy having many disparate tools that do one thing well. .NET on the other hand is far more unified in almost everything (there are pros/cons to both).

      • Adam says :

        A-lot of very valid points!

        I can’t say I’ve found anything missing when running LINQ queries on reactive Observables, the only thing I don’t think I’ve tried is “OrderBy.”

  6. paul says :

    All you have to do is read this to get the real picture…

    http://www.coderanch.com/t/619658/java/java/Comparing-Java

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 1,992 other followers

%d bloggers like this: