Fix: Unexpected row deduplication using eliminate_full_outer_join

[liaco]

The union syntax cause deduplication of rows from source tables, to achieve the intended result but avoid unintended deduplication I changed the logic to a left join + an anti join(right join + not exists) with union all.

This should be more in line with the behaviour of FULL OUTER JOINs since it keeps duplicate rows that would be present if there are duplicate rows in the two joined tables.
https://en.wikipedia.org/wiki/Join_%28SQL%29#Full_outer_join

Other possible ways to achieve this are:

• filtering for NULLS on joining condition columns in the right side of the union
• use a pure anti-join without the right join syntax (this would require table identifiers on columns selected)
• use inner joins + more anti-joins (same requirements as the previous one)

[VaggelisD]

Hey @liaco, good catch and thank you for the PR.

I think the 1st bullet point solution would be a bit easier to comprehend (and possibly faster), I presume it'd look like this. What do you think?

[georgesittas]

Left a couple of comments. I'd prefer a simpler way to implement this, if possible.

[liaco]

I think the 1st bullet point solution would be a bit easier to comprehend (and possibly faster), I presume it'd look like this. What do you think?

I agree, I am not sure however if the code is simpler, it would require something like this to identify cols from the left side table

join_cols = [
    col for col in full_outer_join.args.get("on", exp.Expression()).find_all(exp.Column) 
    if col.table==tables[1]
    ] or [exp.column(identifier, tables[1]) for identifier in full_outer_join.args.get("using")]

and then

.where(exp.Is(this=join_cols[0], expression=sqlglot.exp.null()))

Let me know what you think

[VaggelisD]

@liaco I see, your solution is fine, we'll keep iterating on that.

[liaco]

@georgesittas I implemented your feedback, I don't think is possible to simplify it more due to the nature of full joins: to emulate the functionality we need to use unions or interceptions of available joins (inner, left) to achieve a partial result and then use anti joins to fill in the gaps, however to use anti-joins we need necessarily to identify joining conditions first.

One other way is building a CTE to select from with the union of common joining identifiers and simply left join the two tables (something like this), but it involves more complexity.

Let me know what you think

[georgesittas]

Thanks a lot for the PR @liaco, great work!

[VaggelisD]

One final (minor) comment, thank you for the PR & for the quick iterations.

[liaco]

Thank you for your thoughtful reviews, happy to give my small contribute

[georgesittas]

Check out 7ecd519

[georgesittas]

Thank you for your thoughtful reviews, happy to give my small contribute

Appreciate the contribution and the rapid iterations! :-)