The SQL Language’s Most Missing Feature

SQL is also awesome in many ways. We can write out the most complex truths and facts and have the database tell us the answer in no time.

But the SQL language is arguably the most beautiful programming language out there. It has so many caveats that people like me get ridiculously rich selling consulting services just to merely explain its semantics. One of the best examples of how twisted the SQL language is, is Alex Bolenok’s article about SQL NULL on Tech.Pro.

Now, one of the biggest criticisms of SQL is its verbosity. Most SQL dialects have virtually no constructs to avoid repetition. It is a common use case in SQL to filter, group, select, and order by the same expression. An example:

SELECT   first_name || ' ' || last_name
FROM     customers
WHERE    first_name || ' ' || last_name LIKE 'L% E%'
GROUP BY first_name || ' ' || last_name
ORDER BY first_name || ' ' || last_name

You might write a different query for the same, sure. But even if you didn’t doesn’t it bother you that there is hardly any way to re-use the first_name || ' ' || last_name concatenation expression?

Well, you can at least reuse it in the ORDER BY clause:

SELECT   first_name || ' ' || last_name AS name
FROM     customers
WHERE    first_name || ' ' || last_name LIKE 'L% E%'
GROUP BY first_name || ' ' || last_name
ORDER BY name

And in MySQL, you can also reuse it in the GROUP BY clause, although we think that this is a bad idea:

SELECT   first_name || ' ' || last_name AS name
FROM     customers
WHERE    first_name || ' ' || last_name LIKE 'L% E%'
GROUP BY name
ORDER BY name

Standard SQL solutions for column reuse

Sure, you could create a derived table:

SELECT   name
FROM (
  SELECT first_name || ' ' || last_name
  FROM   customers
) c(name)
WHERE    name LIKE 'L% E%'
GROUP BY name
ORDER BY name

… or a common table expression, which is basically a reusable derived table, or a local view:

WITH c(name) AS (
  SELECT first_name || ' ' || last_name
  FROM   customers
)
SELECT   name
FROM     c
WHERE    name LIKE 'L% E%'
GROUP BY name
ORDER BY name

But why do I even have to create a table to reuse / rename a simple column expression? Why can’t I just simply use something like a common column expression? Like this?

-- Common column expression that can be appended
-- to any table expression.
SELECT   name
FROM     customers
WITH     name AS first_name || ' ' || last_name
WHERE    name LIKE 'L% E%'
GROUP BY name
ORDER BY name

Note that the common column expression would be scoped to the table source. In other words, it only makes sense in the context of a FROM clause and the tables specified in the FROM clause. In a way, the following two expressions would be exactly equivalent:

-- Proposed syntax
FROM     customers
WITH     name AS first_name || ' ' || last_name

-- Existing syntax
FROM (
  SELECT customers.*,
         first_name || ' ' || last_name AS name
  FROM   customers
) AS customers

The proposed syntax could also be applied to joined tables:

-- Proposed syntax
FROM customers
  AS c
WITH name AS c.first_name || ' ' || c.last_name
JOIN addresses
  AS a
WITH address AS a.street || '\n' || a.city
  ON c.address_id = a.address_id
WITH full_address AS c.name || '\n' || a.address

-- Or alternatively, if you don't need to tightly 
-- scope these common column expressions to their
-- corresponding tables:
FROM customers AS c
JOIN addresses AS a
  ON c.address_id = a.address_id
WITH name AS c.first_name || ' ' || c.last_name,
     address AS a.street || '\n' || a.city,
     full_address AS name || '\n' || address

So in plain English, the new WITH clause could be appended to any type of table expression. With parentheses and comments, the first among the above examples would read:

FROM (
  customers AS c
  -- The "name" column is appended to "c"
  WITH name AS c.first_name || ' ' || c.last_name
)
JOIN (
  addresses AS a
  -- The "address" column is appended to "a"
  WITH address AS a.street || '\n' || a.city
) ON c.address_id = a.address_id
-- The "full_address" column is appended to the
-- above joined table expression
WITH full_address AS c.name || '\n' || a.address

The above syntax would then again be exactly equivalent to this:

FROM (
  SELECT 
    c.*, a.*, 
    c.name || '\n' || a.address AS full_address
  FROM (
    SELECT c.*, 
           c.first_name || ' ' || c.last_name
    FROM customers AS c
  ) c
  JOIN (
    SELECT a.*,
           a.street || '\n' || a.city
    FROM addresses AS a
  ) a
  ON c.address_id = a.address_id
)

SQL’s most missing language feature

Since SQL:1999, we luckily have common table expressions – the WITH clause that can be prepended to any SELECT statement (and to other DML statements in some dialects). Using common table expressions, we can avoid repeating commonly used derived tables.

But such a thing is not possible for columns. The one language feature SQL is in most dire need of are thus common column expressions.

MySQL Bad Idea #666

MySQL… We’ve blogged about MySQL before. Many times. We’ve shown bad ideas implemented in MySQL here:

But this beats everything. Check out this Stack Overflow question. It reads: “Why Oracle does not support ‘group by 1,2,3’?”. At first, I thought this user might have been confused because SQL allows for referencing columns by (1-based!) index in ORDER BY clauses:

SELECT first_name, last_name
FROM customers
ORDER BY 1, 2

The above is equivalent to ORDER BY first_name, last_name. The indexes 1, 2 refer to columns from the projection. This might be useful every now and then to avoid repeating complex column expressions, although it is probably a bit risky as you can change ordering semantics when adding a column to the SELECT clause.

But this user wanted to use the same syntax for the GROUP BY clause. And this actually works in MySQL! Check out the following query:

SELECT a, b
FROM (
  SELECT 'a' a, 'b' b, 'c' c UNION ALL
  SELECT 'a'  , 'c'  , 'c'   UNION ALL
  SELECT 'a'  , 'b'  , 'd'
) t
GROUP BY 1, 2
ORDER BY 2

See an example on SQLFiddle

The above yields…

| A | B |
|---|---|
| a | b |
| a | c |

But what would this even mean? According to our in-depth explanation of SQL clauses, the projection (SELECT clause) is logically evaluated after the GROUP BY clause. In other words, the columns defined in the SELECT clause are not yet in scope of the GROUP BY clause. Hence, the only reasonable semantics of column indexes would be the index from the table source t. But that’s not the case. Check out this alternative query:

SELECT a, b
FROM (
  SELECT 'a' a, 'b' b, 'c' c UNION ALL
  SELECT 'a'  , 'c'  , 'c'   UNION ALL
  SELECT 'a'  , 'b'  , 'd'
) t
GROUP BY 1, 2
ORDER BY 2

See an example on SQLFiddle

This now yields:

| B | C |
|---|---|
| b | c |
| c | c |
| b | d |

And it’s (probably?) the expected behaviour in MySQL as the documentation states:

Columns selected for output can be referred to in ORDER BY and GROUP BY clauses using column names, column aliases, or column positions. Column positions are integers and begin with 1

The documentation actually treats GROUP BY in a very similar fashion as ORDER BY. For instance, it is possile to specify the ordering direction using GROUP BY only:

MySQL extends the GROUP BY clause so that you can also specify ASC and DESC after columns named in the clause:

SELECT a, COUNT(b) 
FROM test_table GROUP BY a DESC;

While we cannot figure out a reasonable edge-case that breaks this feature, we still think that there is something fishy about it. The fact that the SELECT clause is logically evaluated after the table source (FROM, WHERE, GROUP BY, HAVING), allowing the GROUP BY clause to reference it seems to lead to a weird understanding of SQL.

On the other hand, SQL is a very verbose language with little support for declaring reusable objects, short of common table expressions and the WINDOW clause. It is actually a bit surprising that the SQL standards folks would support this WINDOW clause to declare reusable window frames before introducing much more usable “common column expressions”, e.g:

-- Common column/table expressions:
WITH x AS CASE t1.a 
          WHEN 1 THEN 'a'
          WHEN 2 THEN 'b'
                 ELSE 'c'
          END, 
     y AS SOME_FUNCTION(t2.a, t2.b)
SELECT x, NVL(y, x)
FROM t1 JOIN t2 ON t1.id = t2.id
GROUP BY x, y
ORDER BY x DESC, y ASC

With common column expressions, reusing column expressions is independent of the SELECT clause itself. In other words, you can reuse column expressions in JOIN clauses, WHERE clauses, GROUP BY clauses, HAVING clauses, etc. without having to actually SELECT them.

So, to be fair with MySQL, while this feature is a non-feature in its current form, it provides a workaround for SQL’s verbosity.