Incorrect handling of null variables in 'where' clause
Problem Description:
The following LINQ-to-Entites query will return the expected results:
var result = from o in table
where o.column == null
select entry;
However, the following query will never return any results:
int? myNullInt = null;
var result = from o in table
where o.column == myNullInt
select entry;
This is because the first query is correctly generating a 'column IS INT' query, while the second one generates a 'column = @p_linq_p; @p_linq_p = NULL'.
This problem affects both LINQ-to-Entities and LINQ-to-SQL. However, while LINQ-to-SQL has the not-so-bad workaround
where o.column.Equals(myNullInt)
The workaround for LINQ-to-Entities is much harder to follow and more painful to write:
where (myNullInt == null ? o.column == null : o.column == myNullInt)
UPDATE: The SQL CE issue has been fixed in the EF6 provider and we changed the default behavior of EF6 to compensate for three-valued logic.
UPDATE: changing the status from completed to started. Turns out that SQL Server Compact provider for EF is incompatible with this feature, therefore we are not making this option the default in DbContext API. The option is still available going through the ObjectContext API.
The option to compensate for three-valued logic used in null equality comparisons performed by SQL databases will be included in the update of the EF core libraries that will ship as part of .NET 4.5. When enabled, this option will cause a LINQ expression such as ‘a == b’ to be translated to an equivalent SQL expression that will evaluate to true when both ‘a’ and ‘b’ are simultaneously null.
Since the option will cause a change in query behavior that could potentially break existing applications and could also have performance impact in specific scenarios, it will be disabled by default for the ObjectContext API. However, the plan is to enable it by default for the DbContext API in the new version of Entity Framework (i.e. the EntityFramework package in NuGet) that we will release alongside .NET 4.5.
We encourage you to add or vote for any other suggestions mentioned in the comments in separate items.
21 comments
-
Paul Anthoney
commented
Now that EF5 and .NET 4.5 are released, is this option available. It doesn't appear to be the default with DbContext?
-
Robby Maddox
commented
@Diego - Thanks for the correction on the result of null==null. I guess it just goes to show that I'm stuck thinking in boolean logic rather than tri-valued logic. ;)
Could you explain the UseCSharpNullComparisonBehavior flag and how it works? When I searched for it online, I couldn't find any useful results. The MSDN documentation says "Gets or sets {insert text here}". Really helpful. lol ;)
Thanks again.
-
AdminDiego Vega
(Admin, DataFx)
commented
@Robby Maddox: good catch on the link. I pasted the admin link, which won't work for most users of the site :)
Regarding tri-valued logic, null==null is actually not false in the database, but null! The question is how to make the database behave in a similar way to in-memory queries. We chose to compensate for the differences, but there are several implications: some queries become more expensive because predicates contain extra terms. We belive we have hit a good compromise with the UseCSharpNullComparisonBehavior flag, but as I said before we are not enabling this by default because it could cause unexpected results. And with "unexpected results" I mean not only unexpected data (I agree with you that the majority of applications won't intentionally depend of the current behavior) but because we have to produce queries with different characteristics when the option is enabled.
-
Robby Maddox
commented
I can't imagine what DESIRABLE existing code would be broken if EF and L2S simply rejected the idea of ANSI nulls in all situations. If there's any EF or L2S code in production that depends on null == null evaluating to false, then it could only be 0.0001% of the EF and L2S code out there. So I say it should be changed with ample warning to the programming community and then that 0.0001% can be changed to something like (a == null ? false : b == null ? false : a == b).
PS: Your link is broken Diego, I think it should be https://data.uservoice.com/forums/72025-ado-net-entity-framework-ef-feature-suggestions/suggestions/2410716-sum-max-min-count-and-average-of-an-e
Thanks
-
AdminDiego Vega
(Admin, DataFx)
commented
Eamon, no that is not covered by this new option. I have created a separate idea for you, please vote for it: https://data.uservoice.com/admin/forums/72025-ado-net-entity-framework-ef-feature-suggestions/suggestions/2410716-sum-max-min-count-and-average-of-an-e.
Thanks
-
Eamon Nerbonne
commented
Will this future .NET 4.5 EF version also fix the fact that Sum() of an empty set should be 0 rather than what it currently is, NULL?
-
Patrick Cummins
commented
Ridiculous. RDBMS's have supported null values for what.... 20, 30 years? Get with the program MS
-
Cemal Erdemir
commented
This bug bite me again.MS please fix this.
-
David Richter commented
Jeez, MS... fix this!
-
Mikey Cee
commented
Agree, it is totally retarded and leaky to override C#'s null comparison semantics with SQL's.
Retarded. Fix it.
-
Anonymous
commented
this is painful! I can't believe it's EF 4.0 and this thing exists!
-
Aaron Murray
commented
I have a nullable nvarchar(256) column and this is a hassle when the string variable is null.
-
Geetha
commented
Thanks a lot! it solves my nullable int column issue!!!!
-
Tommy
commented
This is a real pain in the backside. We're migrating an L2S project over to EF... it was a real nuisance before doing object.equals() but now its on a whole new level.
Find all "Function(n) Object.Equals(", Subfolders, Keep modified files open, Find Results 2, "Entire Solution", "*.vb"
...
Matching lines: 267 Matching files: 89 Total files searched: 1389=/
-
Nick Evans
commented
This is a problem for me as well and results in very inelegant code !
-
Leather
commented
This is causing me a headache too... please address it!
-
AdminDiego Vega
(Admin, DataFx)
commented
Daniel, thanks a lot for the links! This really helps!
-
Daniel.Pflughoeft
commented
I will copy my original argument against this behavior from https://connect.microsoft.com/data/feedback/details/607404/entity-framework-and-linq-to-sql-incorrectly-handling-nullable-variables
"This might make some modicum of sense if you consider that, with ANSI-NULLs enabled, NULL != NULL in SQL. However, when you consider that this *isn't* how nulls work in C#/VB.Net (what people are coding this in), and this also isn't how it works in every other Linq-to-anything provider, it makes no sense to use ANSI-NULLs in Linq2EF and Linq2SQL queries.
In addition, consider that you are using an ANSI-NULL comparison when comparing a variable to a variable, but a C#-null comparison when explicitly comparing a variable to null. Since there is no reason to expect this disparity, or indeed to expect the use of ANSI-NULL comparisons at all outside of joins, it's no wonder this is one of the most common problems relating to Entity Framework on stackoverflow.com:
http://stackoverflow.com/questions/682429/how-can-i-query-for-null-values-in-entity-framework
http://stackoverflow.com/questions/1126133/linq-to-sql-nullable-types-in-where-clause
http://stackoverflow.com/questions/2411894/handling-null-values-in-where-clause-using-linq-to-sql
http://stackoverflow.com/questions/487065/linq-is-weird-or-im-stupid
http://stackoverflow.com/questions/2058616/linq-to-sql-using-int-as-a-parameter-in-a-query
http://stackoverflow.com/questions/586097/compare-nullable-types-in-linq-to-sql
http://stackoverflow.com/questions/629878/weird-linq2sql-let-bugs
http://stackoverflow.com/questions/1960687/entity-framework-mysql-null-comparison
http://stackoverflow.com/questions/2097539/linq-where-column-null-reference-not-the-same-as-column-null
http://stackoverflow.com/questions/3816587/databinding-and-nullable-does-not-seem-to-work-togetherI'm sure there are many more examples on social.msdn.microsoft.com. With this many people running into the same problem, it is clearly not a problem with the people or the documentation but with the framework itself."
-
AdminDiego Vega
(Admin, DataFx)
commented
@Damien: Actually, in theory we could expand the comparison to compensate for server side null equality only if both sides of the equality can be nullable, hence avoiding bloating *all* queries with the expansion. This is not to say we will do this. As Digital-Samurai noted, this would be a breaking change for existing code and adding settings for this represent another challenge, but I think the suggestion stands by its own merits, regardless of implementation concerns.
-
Digital-samurai
commented
I think it is also possible to generate SQL without using ansi_nulls as it is candidate for deprecation. The following snippet accommodates cached scenarios as well
((@Variable is null) AND ([ColumnName] is null))
OR
(@Variable = [ColumnName])Unfortunately this could possibly break existing code, but the inclusion of a setting, perhaps on the model or in the configuration, would allow us to choose this over the current implementation, perhaps we should even call this setting "ansi_nulls" :D