This hilarious article with a click-bait title caught my attention, recently:
A hilarious (although not so true or serious) rant about the current state of JavaScript development in the node ecosystem.
Dependency hell isn’t new
Dependency hell is a term that made it into wikipedia. It defines it as such:
Dependency hell is a colloquial term for the frustration of some software users who have installed software packages which have dependencies on specific versions of other software packages.
The big problem in dependency hell is the fact that small libraries pull in additional depedencies on which they rely in order to avoid too much code duplication. For instance, check out who is using Guava 18.0:
https://mvnrepository.com/artifact/com.google.guava/guava/18.0/usages
You’ll find libraries like:
- com.fasterxml.jackson.core » jackson-databind
- org.springframework » spring-context-support
- org.reflections » reflections
- org.joda » joda-convert
- … 2000 more
Now, let’s assume you’re still using Guava 15.0 in your application. You want to upgrade Spring. Will your application still work? Is that upgrade binary compatible? Will Spring guarantee this to you, or are you on your own? Now what if Spring also uses Joda Time, which in turn also uses Guava? Does this even work? Conversely, can Guava ever depend on Joda Time or will a circular Maven Central dependency cause all singularities to collapse into one huge black hole?
Truth is: You don’t need the dependency
… and by you, I don’t mean the end user who writes complex enterprise applications with Guava (or whatever). You need it. But YOU, dear library developer. You certainly don’t need any dependency.
An example from jOOQ. Being a SQL string manipulation library, we pulled in a dependency on Apache Commons Lang because:
- They have some nice StringUtils, which we like to use
- Their license is also ASL 2.0, which is compatible to jOOQ’s license
But instead of hard wiring a jOOQ 3.x to commons-lang 2.x dependency, we opted for internalising one of their classes and only parts of it, repackaging it as org.jooq.tools.StringUtils. Essentially, we needed things like:
- abbreviate()
- isEmpty()
- isBlank()
- leftPad() (hello node developers)
… and some more. That certainly doesn’t justify pulling in the entire dependency, does it? Because while it wouldn’t matter to us, it would matter to our thousands of users, who might prefer to use an older or newer version of commons-lang. And that’s just the beginning. What if commons-lang had transitive dependencies? Etc.
Please, library developers, avoid dependencies
So, please, dear library developers. Please avoid adding dependencies to your libraries. The only things you should depend on are:
- The JDK
- Some API governed by a JSR (e.g. JPA)
That’s it. We can all start writing better software and stop downloading the whole internet if YOU the library developers start being reasonable and stop being lazy.
Exceptions to the above rules:
- Framework and platform vendors (e.g. Spring, Java EE) are excluded. They define the whole platform, i.e. they impose a set of well-documented dependencies. If you’re using a framework / platform in your application, then you have to abide to the platform’s rules
That’s all. Small libraries like jOOQ must not have any dependency.
Note that jackson-databind uses Guava only in test scope. And for spring-context-support and joda-convert Guava is only an optional dependency.
Thanks for the hint. The test scope is fine, indeed. I should’ve checked that first.
The optional dependency is treacherous, though, because it is still a dependency on API that might not be entirely compatible between versions…
Say more about “internalising one of their classes” instead. You literally copy the relevant classes and put them in new packages of your own? Are there license issues? What if they in turn have dependencies?
I can’t decide if this advice is extremely wise, or complete rubbish. Or both. :-) Clearly there’s a point at which this approach just falls over, as the library (and its needs) get bigger. But I certainly “feel the burn” (heh) of dependency hell (we call it “ivy poisoning”).
Yes. Also, some methods are removed if they’re not needed. For example: https://www.jooq.org/javadoc/latest/org/jooq/tools/StringUtils.html
Not if the internalised dependency is ASL 2.0. Of course, that particular class is not dual-licensed (it cannot be) and copyrights remain where they are. Also, the dependencies and respective licenses are documented on the website: https://www.jooq.org/legal/licensing
They’re removed or inlined.
This approach doesn’t fall over. A library should do one thing and one thing only. Its needs cannot possibly get much bigger. Otherwise, it might turn into a framework or platform, for which there are different rules (e.g. Spring). I’ve mentioned this in the article. If you’re using Spring, you’re bound by their rules and dependencies.
For the truly lazy you can also use the Maven shade plugin. I believe proguard has some similar features as well.
Another reason not to depend on a ton of jars is jar/class scanning. The Servlet +3.0 spec is ambiguous on the loading the of web fragments (classes with annotations). Consequently Tomcat typically has to scan all your jars at boot (even if you select the system properties to disable scanning if there is a single service loader context initializer found (usually from spring web) all jars will be scanned.
Yes, indeed, the shade plugin can be helpful in these cases, but I think it will often load too much, e.g. when you need only 3-4 methods as in jOOQ’s case.
Oh wow, nice point about the jar/class scanning. I hadn’t thought of this, but indeed. Less (useless) code is always better, I guess!
Java 9 Jigsaw can help with this problem with Layers. I have played with layers at https://github.com/torstenwerner/java-9-no-jar-hell and you won’t believe what happened! ;)
A question was raised about the legality of including code from open-source libraries in a closed-source product. (I saw in the auto-generated email, but oddly enough, not here, at least not yet.)
A simple answer (that fulfills the requirements of the vast majority of open-source licenses) is to simply publish (on your company or personal website) your modified version of the open-source library. (The OP is suggesting changing the package names, and removing everything not needed. That counts as modification, even if it’s trivial.)
Oh, you got an E-Mail about that question? I had deleted it as spam, it linked to a completely irrelevant Java blog post with tons of advertisement…
I guess it’s simple. Copyleft: Don’t do what I suggested in the article. Don’t even use copyleft stuff. Permissive: Give the credit that is asked for (and the sources of the modification, etc.) and you’re fine.
Hmm. Usually we find good points of agreement, but on this one (while minor in the larger scope of the original question) I have to differ. If you really need something that is “copylefted” (meaning: you can’t just copy and attribute, e.g. something under the LGPL license), you certainly can use it, you just have to jump through an additional (pretty simple) hoop as described above.
Lukas, if you don’t like copyleft, that’s fine. But please don’t conflate that opinion with a practical solution to a practical problem. (E.g. *assuming* a case where I really need a copylefted library, how do I follow the rules?).
I was thinking of the GPL, not LGPL. Should’ve mentioned that, sorry.
Ah, thanks. That makes sense of why we were seeing the issue differently. Cheers.
How do you keep track when the dependency that you internalized gets updated with a bug fix for the parts you internalized?
Thus far, this hasn’t been an issue for jOOQ, as the internalised libraries were simple enough to be quite feature complete and bug free (JSON and CSV parsers), or they’re part of the jOO* family.