Ever since Martin Fowler’s talks about fluent interfaces, people have started chaining methods all over the place, creating fluent API’s (or DSLs) for every possible use case. In principle, almost every type of DSL can be mapped to Java. Let’s have a look at how this can be done
DSL rules
DSLs (Domain Specific Languages) are usually built up from rules that roughly look like these
1. SINGLE-WORD
2. PARAMETERISED-WORD parameter
3. WORD1 [ OPTIONAL-WORD ]
4. WORD2 { WORD-CHOICE-A | WORD-CHOICE-B }
5. WORD3 [ , WORD3 ... ]
Alternatively, you could also declare your grammar like this (as supported by this nice Railroad Diagrams site):
Grammar ::= (
'SINGLE-WORD' |
'PARAMETERISED-WORD' '('[A-Z]+')' |
'WORD1' 'OPTIONAL-WORD'? |
'WORD2' ( 'WORD-CHOICE-A' | 'WORD-CHOICE-B' ) |
'WORD3'+
)
Put in words, you have a start condition or state, from which you can choose some of your languages’ words before reaching an end condition or state. It’s like a state-machine, and can thus be drawn in a picture like this:
Java implementation of those rules
With Java interfaces, it is quite simple to model the above DSL. In essence, you have to follow these transformation rules:
- Every DSL “keyword” becomes a Java method
- Every DSL “connection” becomes an interface
- When you have a “mandatory” choice (you can’t skip the next keyword), every keyword of that choice is a method in the current interface. If only one keyword is possible, then there is only one method
- When you have an “optional” keyword, the current interface extends the next one (with all its keywords / methods)
- When you have a “repetition” of keywords, the method representing the repeatable keyword returns the interface itself, instead of the next interface
- Every DSL subdefinition becomes a parameter. This will allow for recursiveness
Note, it is possible to model the above DSL with classes instead of interfaces, as well. But as soon as you want to reuse similar keywords, multiple inheritance of methods may come in very handy and you might just be better off with interfaces.
With these rules set up, you can repeat them at will to create DSLs of arbitrary complexity, like jOOQ. Of course, you’ll have to somehow implement all the interfaces, but that’s another story.
Here’s how the above rules are translated to Java:
// Initial interface, entry point of the DSL
// Depending on your DSL's nature, this can also be a class with static
// methods which can be static imported making your DSL even more fluent
interface Start {
End singleWord();
End parameterisedWord(String parameter);
Intermediate1 word1();
Intermediate2 word2();
Intermediate3 word3();
}
// Terminating interface, might also contain methods like execute();
interface End {
void end();
}
// Intermediate DSL "step" extending the interface that is returned
// by optionalWord(), to make that method "optional"
interface Intermediate1 extends End {
End optionalWord();
}
// Intermediate DSL "step" providing several choices (similar to Start)
interface Intermediate2 {
End wordChoiceA();
End wordChoiceB();
}
// Intermediate interface returning itself on word3(), in order to allow
// for repetitions. Repetitions can be ended any time because this
// interface extends End
interface Intermediate3 extends End {
Intermediate3 word3();
}
With the above grammar defined, we can now use this DSL directly in Java. Here are all the possible constructs:
Start start = // ...
start.singleWord().end();
start.parameterisedWord("abc").end();
start.word1().end();
start.word1().optionalWord().end();
start.word2().wordChoiceA().end();
start.word2().wordChoiceB().end();
start.word3().end();
start.word3().word3().end();
start.word3().word3().word3().end();
And the best thing is, your DSL compiles directly in Java! You get a free parser. You can also re-use this DSL in Scala (or Groovy) using the same notation, or a slightly different one in Scala, omitting dots “.” and parentheses “()”:
val start = // ...
(start singleWord) end;
(start parameterisedWord "abc") end;
(start word1) end;
((start word1) optionalWord) end;
((start word2) wordChoiceA) end;
((start word2) wordChoiceB) end;
(start word3) end;
((start word3) word3) end;
(((start word3) word3) word3) end;
Real world examples
Some real world examples can be seen all across the jOOQ documentation and code base. Here’s an extract from a previous post of a rather complex SQL query created with jOOQ:
create().select(
r1.ROUTINE_NAME,
r1.SPECIFIC_NAME,
decode()
.when(exists(create()
.selectOne()
.from(PARAMETERS)
.where(PARAMETERS.SPECIFIC_SCHEMA.equal(r1.SPECIFIC_SCHEMA))
.and(PARAMETERS.SPECIFIC_NAME.equal(r1.SPECIFIC_NAME))
.and(upper(PARAMETERS.PARAMETER_MODE).notEqual("IN"))),
val("void"))
.otherwise(r1.DATA_TYPE).as("data_type"),
r1.NUMERIC_PRECISION,
r1.NUMERIC_SCALE,
r1.TYPE_UDT_NAME,
decode().when(
exists(
create().selectOne()
.from(r2)
.where(r2.ROUTINE_SCHEMA.equal(getSchemaName()))
.and(r2.ROUTINE_NAME.equal(r1.ROUTINE_NAME))
.and(r2.SPECIFIC_NAME.notEqual(r1.SPECIFIC_NAME))),
create().select(count())
.from(r2)
.where(r2.ROUTINE_SCHEMA.equal(getSchemaName()))
.and(r2.ROUTINE_NAME.equal(r1.ROUTINE_NAME))
.and(r2.SPECIFIC_NAME.lessOrEqual(r1.SPECIFIC_NAME)).asField())
.as("overload"))
.from(r1)
.where(r1.ROUTINE_SCHEMA.equal(getSchemaName()))
.orderBy(r1.ROUTINE_NAME.asc())
.fetch()
Here’s another example from a library that looks quite appealing to me. It’s called jRTF and it’s used to create RTF documents in Java in a fluent style:
rtf()
.header(
color( 0xff, 0, 0 ).at( 0 ),
color( 0, 0xff, 0 ).at( 1 ),
color( 0, 0, 0xff ).at( 2 ),
font( "Calibri" ).at( 0 ) )
.section(
p( font( 1, "Second paragraph" ) ),
p( color( 1, "green" ) )
)
).out( out );
Summary
Fluent APIs have been a hype for the last 7 years. Martin Fowler has become a heavily-cited man and gets most of the credits, even if fluent APIs were there before. One of Java’s oldest “fluent APIs” can be seen in java.lang.StringBuffer, which allows for appending arbitrary objects to a String. But the biggest benefit of a fluent API is its ability to easily map “external DSLs” into Java and implement them as “internal DSLs” of arbitrary complexity.
Although is possible to make everything with Java, it is not the proper tool for creating DSLs.
Groovy let you create very human-readable domain languages:
http://gfrison.com/post/12963823561/dsl-pills-with-groovy
I couldn’t agree more. Other languages than Java have made big steps ahead in allowing for custom internal DSL’s. C# also has great extension capabilities that were introduced in order to allow for LINQ to be created.
But unfortunately, for many of us, in running projects, using Groovy (or Scala, etc) is not an option (yet).
Good stuff. Fluent interfaces really drive home the point that good API design is important. Thanks for sharing.
I found the example illustrative. and the formalization with the BNF grammar and how it maps to Java language constructs helpful.
Thank you. I might actually further formalise this with a source code generator for fluent API’s in the near future.
Very nice post… It’s a really simple and it’s an effective summary!
I propose some other point, if it’s possible to expand the discussion:
– What about static method?
– What about ‘local context’ (context simulation with threadlocal)? (in chaining i lost some return values, sometimes localcontext it’s needed to recover this values)
– We must stop to only chaining o we can try other approach?
– What about Builders and Factories?
All are ingredients about internal DSL…
Thank You, and again, Compliments!
Extending the discussion is always welcome!
– I think I mentioned static methods. They’re quite useful as entry points to a DSL, especially when combined with static imports
– Context / Builders / Factories are implementation-specific problems. I tend to solve this either by avoiding static methods (and create the whole DSL-generated objects from a factory), or by passing a context to a “render” method at the end of a DSL-generated object’s lifecycle. But ThreadLocal is definitely a suitable idea.
As I told Ferenc, I think I’m going to write a general prototype for transforming BNF definitions to Java classes (including implementation stubs). This could be quite an interesting project!
Have the BNF definition be written in Java, using a fluent api! :D
This is an interesting post.
As others have pointed out, languages such as Groovy or Scala would be more appropriate for creating internal DSLs. I do however like the approach that you’ve taken to formalize the grammar and map it to Java interfaces. This is a nice recipe that can easily be reproduced.
Thanks.
Thank you. I do think this can be useful to others every now and then…
Cheers
Lukas
I have seen several remarks how other languages do better at creating DSLs than Java.
Is there some overview of concrete advantages of other languages in this respect?
I’m coming from the idea that while making a DSL in Java is a lot of useless clutter, *using* a DSL in Java should not be more complicated than elsewhere. That’s just a thought I’m toying with, I may well be wrong – so I’d be interested in arguments both for and against.
I’m not aware of any concrete overview. If there is, feel free to share it here, I’d be interested.
The claim that other languages (in particular Scala and Groovy) are better is based on the facts that
– Scala allows for many more features (e.g. operator overloading, case classes, traits, etc)
– Groovy has native DSL creation support.
In general you could say that every language allows for creating internal DSLs but the DSL will only ever be as expressive as the host language allows it to be. In that way, an internal DSL in Java is probably less expressive and thus less powerful than an internal DSL in Scala / Groovy.
Note that this article is about internal DSLs (DSLs built within the scope of another language). If you write an external DSL (DSLs that are parsed / compiled / interpreted), you don’t have any limits whatsoever in terms of expressivity, no matter what platform you use to access that DSL. However, building expressions for such a DSL will probably mean resorting to string concatenation, then.
Just internal DSLs.
External DSLs run into all kinds of trouble, from limited tool support (no code complete for your FooScript, sorry) to impedance mismatches (SQL is a case in point).
I have started thinking about making DSL types generic so that they can accept semantic actions. That would make the DSL the parser and the semantic actions the code generator (or code transformer or analysis tool or whatever you want it to be).
Heck, you could build an EBNF DSL that generates Java classes. The Java plugin would then generate .java files, the Bytecode plugin runtime classes, the dump plugin would emit stuff like FIRST and FOLLOW sets and/or do other language designy things. Iti would sure remove the impedance mismatch between an EBNF-to-Java compiler and Java code :-)
Don’t have any running code though, and may never have – too many other things in the pipeline. Just tossing around ideas here :-)
Re how other languages are better than Java: I’d really like to see concrete examples where Java sucks for making and/or using DSLs.
I have a hunch that most of the problems can be eliminated. In fact the above article is a nice example how to deal with quite a lot of shortcomings!
The really interesting point being what remains, what are the problems where a Java-based DSL would hit a brick wall. Where’s the limit?
Yes, I had been thinking about creating a class generator in order to auto-generate jOOQ’s DSL API. This would be particularly useful when jOOQ users want to create their own APIs. Some examples:
– MySQL users could strip all Oracle features from the API
– They could re-define their jOOQ “keywords” the way they prefer, such as GROUP_BY() instead of groupBy()
But… just as you, I probably don’t have the time to do that. :)
About Java “sucking” at being a host for DSLs: I didn’t say that. I just said that Scala in particular would be more powerful. Java has a lot of issues with generics, covariance, etc. I could imagine (but I don’t have any experience), that a Scala DSL could be more expressive because Scala has a few more features itself, compared to Java (such as operator overloading). With jOOQ, I have to write
COLUMN_A.add(COLUMN_B).div(COLUMN_C)
With Scala, I could write
(COLUMN_A + COLUMN_B) / COLUMN_C
That’s all I’m saying. When I have some more time, I’ll explore into re-writing the jOOQ API for Scala, taking full advantage of Scala features. Here are two examples:
– jOOQ as it is today in Scala: https://blog.jooq.org/2011/12/11/the-ultimate-sql-dsl-jooq-in-scala/
– An example of what jOOQ could be in Scala: https://blog.jooq.org/2011/12/24/sql-in-scala-where-jooq-could-go/
Fluent interface calculation example for formula:
can be implemented following way:
Result result1, result2;
Calculation.startWith(B).substractBy(C) .multiplyBy(A).toAbsoluteValue() .rememberAsResult(result1); Calculation.startWith(E).substractBy(F) .rememberAsResult(result2); When(result2).isNotZero().thenStartWith(D) .divideBy(result2).add(result1) .roundToTwoDecimals().isEqualTo(R); When(result2).isZero().thenStartWith(result1) .roundToTwoDecimals().isEqualTo(R);I generaly prefer the use of methods instead of operators that scala may add or overwrite.This is because they tend to be more explicit and thus require less time to learn as illustrated just above. And if the standard mathematic operator help to understand, the Fluent API used with your favorite java editor context-suggest the right methods pefectly.
With Scala, overloaded operators are just methods too. Hence, a good IDE should be capable of context-suggesting operators just like any other methods (including Javadocs!).
Platforms other than Scala may not have found such an elegant solution to operator overloading, in case of which I agree with you.
Readability, of course, is a matter of taste, I agree. An API designer may still choose to provide every functionality twice. Once as an operator, and once as a method with a concise name.
The tricky thing I have found with fluent API design is if it involves multiple states (ie more than just builders like StringBuilder) it becomes rather difficult to document with out continuously showing full examples.
That is in my opinion Javadoc on individual classes and methods becomes less and less useful for documenting multi-state fluent API’s such as jOOQ. Browsing through the Javadoc may even confuse with myriads of interfaces and inheritance that are being used to help support the fluent interface. Thus I find separate example/unit test doc best for fluent API.
The other issue with multi-state fluent API’s is that you can forget to execute that last method on the chain (#end() in your examples) that actually does something. I have had this happen in my own fluent API design and the only way I have combated it with is consistent naming of an “end” method that actually executes.
I share your documentation point of view. The best way to document a BNF-based DSL is by using a BNF notation (as depicted in this post). Javadoc can only summarise a small portion of the internalised domain specific language. (note that adding extensive BNF examples to the jOOQ manual is on the roadmap).
Your second issue is also an interesting point. I’ve had this happen to myself, when writing jOOQ unit tests. jOOQ’s execute() and fetch() methods aren’t 100% intuitive as everything preceding those method looks like SQL, while execute() and fetch() do not…
great!
I’d like to let you know that even years after you wrote this article, there are still people who find it and feel really astonished how expressive Java can be – if used appropriately. I feel that this interface design is not sensible for each and every builder or factory, but for certain central and vastly spread classes in your system, this can really be a help for correct usage. The fine thing is that you enforce more semantics as it possible with “flat” builders alone. A user of a class creator defined this way is way better lead through the construction process as he or she simply has no possiblility to combine parameters wrongly or forget crucial information. I am very impressed. Thanks for having shared this – even if it’s 3 3/4 years old. :-)
Thanks for your nice words, Dirk! It is one of our more popular articles, indeed.
I agree. For something as simple as a
StringBuilder, simple method-chaining is certainly enough. There is not state machine or BNF notation or anything involved.Our use-case, however, is to model all of the SQL language in Java (https://www.jooq.org), and this article only scratches the surface of what we’re doing – yet it explains the essence. An internal DSL can be specified via a BNF and an API can implement that BNF.
Not sure if you’ve seen this project: fluflu by Peter Verhas: https://github.com/verhas/fluflu. He created a simple annotation processor that generates the API from the implementation. Very neat! All you have to write is the backing implementation and the annotation-encoded BNF.
Hope this helps!
Lukas
I have developed a fluent wrapper API for the Eclipse databinding framework using techniques described in this article.
The point of the API is to bind an observable value to another observable value. Values can be single values, lists or sets. The API allows users to set converters, validators and other properties of the binding. It tries to give as much type safety as possible, and only make relevant operations available in all steps.
The API consists of 31 different interfaces, defining the fluent pipeline steps.
Currently the code can be found here:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=552667#c35
Example code:
Bind.twoWay() // 1 .from(modelObservable) // 2 .validateAfterConvert(modelValidator) // 3 .convertTo(IConverter.create(i -> Objects.toString(i, ""))) // 4 .convertFrom(IConverter.create(s -> s.isEmpty() ? 0 : Integer.decode(s))) // .to(WidgetProperties.text(SWT.Modify).observe(text)) // 5 .validateTwoWay(widgetValidator) // 6 .bind(bindingContext); // 7Explanation of the example:
1. First the user chooses between a two-way or one-way binding. The binding direction (model-to-target or target-to-model) can also be chosen.
2. The from-end observable is given. Here the API chooses between value, list of set bindings.
3. The from-end is configured. This involves setting validators and binding policies (convert-only, only-on-request). Only methods that are relevant for two-way or one-way bindings are present for the respective cases.
4. Converters are set. Here the to-end observable gets its type. The API ensures that two converters are set for two-way bindings.
5. The to-end observable is set.
6. The to-end is configured, in the same was as the from-end.
7. The bind-method is called, with a DataBindingContext as argument. This internally creates UpdateValueStrategy objects and calls DataBindingContext#bindValue.
Thanks for sharing. I wonder if this approach is worth it in your case? In jOOQ’s case, an external DSL is being mimicked, whose syntax can be described using a BNF, hence the BNF-style internal DSL (producing an excessive amount of API) makes sense. For an ordinary builder, is it worth it? Is there really a “one true path” or “one true syntax” and order of method calls?
I though you as a fellow API design enthusiast might find this interesting, Lukas. :)
> I wonder if this approach is worth it in your case?
Yeah, I agree, that’s an important question! The resulting API really turned out to pretty large and complex, and its not clear to me whether it is worth it.
The “true way through the API” isn’t as clear as in jOOQ, but there is value in the fluent API approach, compared to a simple builder. It guides and protects users from errors more.
The API works like this:
* In the first step the users decides between a one-way or two-way binding.
* Then the from-end observable is set, fixating its type.
* The from-end type is used to ensure that validators of the right type are used.
* Other from-end properties are set.
* Then the user has a choice about how to fixate the type of the to-end observable:
– Either a to-end observable of the SAME type as the from-end is given.
– Or converters are first set, then the to-end observable of a different type can be set.
* The to-end validators are set, using the to-end type.
* Other to-end properties are set.
The converters, validators and other settings that have to made are different between one-way and two-way bindings, and also for bindings between values, lists and sets. This is all captured in the API.
The result turned out to be more complicated than I expected. I have not yet decided if I should use the current version, or try to simplify it.