Guard against incoming rowcount estimate of NaN in cost_mergejoin().

Although rowcount estimates really ought not be NaN, a bug elsewhere
could perhaps result in that, and that would cause Assert failure in
cost_mergejoin, which I believe to be the explanation for bug #5977 from
Anton Kuznetsov.  Seems like a good idea to expend a couple more cycles
to prevent that, even though the real bug is elsewhere.  Not back-patching,
though, because we don't encourage running production systems with
Asserts on.

src/backend/optimizer/path/costsize.c

@@ -1704,10 +1704,10 @@ cost_mergejoin(MergePath *path, PlannerInfo *root, SpecialJoinInfo *sjinfo)
 				innerendsel;
 	Path		sort_path;		/* dummy for result of cost_sort */
 
-	/* Protect some assumptions below that rowcounts aren't zero */
-	if (outer_path_rows <= 0)
+	/* Protect some assumptions below that rowcounts aren't zero or NaN */
+	if (outer_path_rows <= 0 || isnan(outer_path_rows))
 		outer_path_rows = 1;
-	if (inner_path_rows <= 0)
+	if (inner_path_rows <= 0 || isnan(inner_path_rows))
 		inner_path_rows = 1;
 
 	if (!enable_mergejoin)