(Don't forget that an alias is required.)  Views reimplemented as expanding
to subselect-in-FROM.  Grouping, aggregates, DISTINCT in views actually
work now (he says optimistically).  No UNION support in subselects/views
yet, but I have some ideas about that.  Rule-related permissions checking
moved out of rewriter and into executor.
INITDB REQUIRED!

ends to clean up (see my message of same date to pghackers), but mostly
it works.  INITDB REQUIRED!

multiple times in the parsetree (can happen in COALESCE or BETWEEN
contexts, for example).  This is a pretty grotty solution --- it will
do for now, but perhaps we can do better when we redesign querytrees.
What we need is a consistent policy about whether querytrees should be
considered read-only structures or not ...

memory contexts.  Currently, only leaks in expressions executed as
quals or projections are handled.  Clean up some old dead cruft in
executor while at it --- unused fields in state nodes, that sort of thing.

materialized tupleset is small enough) instead of a temporary relation.
This was something I was thinking of doing anyway for performance, and Jan
says he needs it for TOAST because he doesn't want to cope with toasting
noname relations.  With this change, the 'noname table' support in heap.c
is dead code, and I have accordingly removed it.  Also clean up 'noname'
plan handling in planner --- nonames are either sort or materialize plans,
and it seems less confusing to handle them separately under those names.

contained a sub-SELECT nested within an AND/OR tree that cnfify()
thought it should rearrange.  Same physical sub-SELECT node could
end up linked into multiple places in resulting expression tree.
This is harmless for most node types, but not for SubLink.
Repair bug by making physical copies of subexpressions that get
logically duplicated by cnfify().  Also, tweak the heuristic that
decides whether it's a good idea to do cnfify() --- we don't really
want that to happen when it would cause multiple copies of a subselect
to be generated, I think.

WHERE in a place where it can be part of a nestloop inner indexqual.
As the code stood, it put the same physical sub-Plan node into both
indxqual and indxqualorig of the IndexScan plan node.  That confused
later processing in the optimizer (which expected that tracing the
subPlan list would visit each subplan node exactly once), and would
probably have blown up in the executor if the planner hadn't choked first.
Fix by making the 'fixed' indexqual be a complete deep copy of the
original indexqual, rather than trying to share nodes below the topmost
operator node.  This had further ramifications though, because we were
making the aforesaid list of sub-Plan nodes during SS_process_sublinks
which is run before construction of the 'fixed' indexqual, meaning that
the copy of the sub-Plan didn't show up in that list.  Fix by rearranging
logic so that the sub-Plan list is built by the final set_plan_references
pass, not in SS_process_sublinks.  This may sound like a mess, but it's
actually a good deal cleaner now than it was before, because we are no
longer dependent on the assumption that planning will never make a copy
of a sub-Plan node.

to simplify constant expressions and expand SubLink nodes into SubPlans
is done in a separate routine subquery_planner() that calls union_planner().
We formerly did most of this work in query_planner(), but that's the
wrong place because it may never see the real targetlist.  Splitting
union_planner into two routines also allows us to avoid redundant work
when union_planner is invoked recursively for UNION and inheritance
cases.  Upshot is that it is now possible to do something like
select float8(count(*)) / (select count(*) from int4_tbl)  from int4_tbl
group by f1;
which has never worked before.

running gcc and HP's cc with warnings cranked way up.  Signed vs unsigned
comparisons, routines declared static and then defined not-static,
that kind of thing.  Tedious, but perhaps useful...

of tuples we are going to retrieve from a sub-SELECT.  Must have been
half asleep when I did this code the first time :-(

subplan: do it if subplan has subplans itself, and always do it if the
subplan is an indexscan.  (I originally set it to materialize an indexscan
only if the indexqual is fairly selective, but I dunno what I was
thinking ... an unselective indexscan is still expensive ...)

looks like it will save computation to do so.

accesses versus sequential accesses, a (very crude) estimate of the
effects of caching on random page accesses, and cost to evaluate WHERE-
clause expressions.  Export critical parameters for this model as SET
variables.  Also, create SET variables for the planner's enable flags
(enable_seqscan, enable_indexscan, etc) so that these can be controlled
more conveniently than via PGOPTIONS.

Planner now estimates both startup cost (cost before retrieving
first tuple) and total cost of each path, so it can optimize queries
with LIMIT on a reasonable basis by interpolating between these costs.
Same facility is a win for EXISTS(...) subqueries and some other cases.

Redesign pathkey representation to achieve a major speedup in planning
(I saw as much as 5X on a 10-way join); also minor changes in planner
to reduce memory consumption by recycling discarded Path nodes and
not constructing unnecessary lists.

Minor cleanups to display more-plausible costs in some cases in
EXPLAIN output.

Initdb forced by change in interface to index cost estimation
functions.

  * Portions Copyright (c) 1996-2000, PostgreSQL, Inc

to all files copyright Regents of Berkeley.  Man, that's a lot of files.

subselects can only appear on the righthand side of a binary operator.
That's still true for quantified predicates like x = ANY (SELECT ...),
but a subselect that delivers a single result can now appear anywhere
in an expression.  This is implemented by changing EXPR_SUBLINK sublinks
to represent just the (SELECT ...) expression, without any 'left hand
side' or combining operator --- so they're now more like EXISTS_SUBLINK.
To handle the case of '(x, y, z) = (SELECT ...)', I added a new sublink
type MULTIEXPR_SUBLINK, which acts just like EXPR_SUBLINK used to.
But the grammar will only generate one for a multiple-left-hand-side
row expression.

documented intepretation of the lefthand and oper fields.  Fix a number of
obscure problems while at it --- for example, the old code failed if the parser
decided to insert a type-coercion function just below the operator of a
SubLink.
CAUTION: this will break stored rules that contain subplans.  You may
need to initdb.

and fix_opids processing to a single recursive pass over the plan tree
executed at the very tail end of planning, rather than haphazardly here
and there at different places.  Now that tlist Vars do not get modified
until the very end, it's possible to get rid of the klugy var_equal and
match_varid partial-matching routines, and just use plain equal()
throughout the optimizer.  This is a step towards allowing merge and
hash joins to be done on expressions instead of only Vars ...

sort order down into planner, instead of handling it only at the very top
level of the planner.  This fixes many things.  An explicit sort is now
avoided if there is a cheaper alternative (typically an indexscan) not
only for ORDER BY, but also for the internal sort of GROUP BY.  It works
even when there is no other reason (such as a WHERE condition) to consider
the indexscan.  It works for indexes on functions.  It works for indexes
on functions, backwards.  It's just so cool...

CAUTION: I have changed the representation of SortClause nodes, therefore
THIS UPDATE BREAKS STORED RULES.  You will need to initdb.

indeed some of 'em were missing support for more node types than that...

sure if they are all fixed, because rewriter is now the stumbling block,
but at least some cases work that did not work before.

real affect now.

_copyResult didn't copy subPlan structure completely.  _copyAgg is still
busted, apparently because of changes from EXCEPT/INTERSECT patch
(get_agg_tlist_references is no longer sufficient to find all aggregates).
No time to look at that tonight, however.

INTERSECT and EXCEPT is available for postgresql-v6.4!

The patch against v6.4 is included at the end of the current text
(in uuencoded form!)

I also included the text of my Master's Thesis. (a postscript
version). I hope that you find something of it useful and would be
happy if parts of it find their way into the PostgreSQL documentation
project (If so, tell me, then I send the sources of the document!)

The contents of the document are:
  -) The first chapter might be of less interest as it gives only an
     overview on SQL.

  -) The second chapter gives a description on much of PostgreSQL's
     features (like user defined types etc. and how to use these features)

  -) The third chapter starts with an overview of PostgreSQL's internal
     structure with focus on the stages a query has to pass (i.e. parser,
     planner/optimizer, executor). Then a detailed description of the
     implementation of the Having clause and the Intersect/Except logic is
     given.

Originally I worked on v6.3.2 but never found time enough to prepare
and post a patch. Now I applied the changes to v6.4 to get Intersect
and Except working with the new version. Chapter 3 of my documentation
deals with the changes against v6.3.2, so keep that in mind when
comparing the parts of the code printed there with the patched sources
of v6.4.

Here are some remarks on the patch. There are some things that have
still to be done but at the moment I don't have time to do them
myself. (I'm doing my military service at the moment) Sorry for that
:-(

-) I used a rewrite technique for the implementation of the Except/Intersect
   logic which rewrites the query to a semantically equivalent query before
   it is handed to the rewrite system (for views, rules etc.), planner,
   executor etc.

-) In v6.3.2 the types of the attributes of two select statements
   connected by the UNION keyword had to match 100%. In v6.4 the types
   only need to be familiar (i.e. int and float can be mixed). Since this
   feature did not exist when I worked on Intersect/Except it
   does not work correctly for Except/Intersect queries WHEN USED IN
   COMBINATION WITH UNIONS! (i.e. sometimes the wrong type is used for the
   resulting table. This is because until now the types of the attributes of
   the first select statement have been used for the resulting table.
   When Intersects and/or Excepts are used in combination with Unions it
   might happen, that the first select statement of the original query
   appears at another position in the query which will be executed. The reason
   for this is the technique used for the implementation of
   Except/Intersect which does a query rewrite!)
   NOTE: It is NOT broken for pure UNION queries and pure INTERSECT/EXCEPT
         queries!!!

-) I had to add the field intersect_clause to some data structures
   but did not find time to implement printfuncs for the new field.
   This does NOT break the debug modes but when an Except/Intersect
   is used the query debug output will be the already rewritten query.

-) Massive changes to the grammar rules for SELECT and INSERT statements
   have been necessary (see comments in gram.y and documentation for
   deatails) in order to be able to use mixed queries like
   (SELECT ... UNION (SELECT ... EXCEPT SELECT)) INTERSECT SELECT...;

-) When using UNION/EXCEPT/INTERSECT you will get:
   NOTICE: equal: "Don't know if nodes of type xxx are equal".
   I did not have  time to add comparsion support for all the needed nodes,
   but the default behaviour of the function equal met my requirements.
   I did not dare to supress this message!

   That's the reason why the regression test for union will fail: These
   messages are also included in the union.out file!

-) Somebody of you changed the union_planner() function for v6.4
   (I copied the targetlist to new_tlist and that was removed and
   replaced by a cleanup of the original targetlist). These chnages
   violated some having queries executed against views so I changed
   it back again. I did not have time to examine the differences between the
   two versions but now it works :-)
   If you want to find out, try the file queries/view_having.sql on
   both versions and compare the results . Two queries won't produce a
   correct result with your version.

regards

    Stefan

[AC_MSG_RESULT(yes) AC_DEFINE(HAVE_LONG_INT_64)],

this line produces something like:

  echo "$ac_t""yes" 1>&6 cat >> confdefs.h <<\EOF

and would append garbage "yes cat" to confdefs.h. Of course the
result confdefs.h is not syntactically correct therefore following
tests using confdefs.h would all fail.  To avoid the problem, we
could switch the order of AC_MSG_RESULT and AC_DEFINE (see attached
patch). This happend on my LinuxPPC box.


Tatsuo Ishii t-ishii@sra.co.jp

Everyone using an [NOT] EXISTS subquery will have noticed that
already.

The bug is in "subselect.c" in the function "SS_process_sublinks()".

Here the whole function as it *SHOULD BE*:

Stephan

   now. Here some tested features, (examples included in the patch):

1.1) Subselects in the having clause 1.2) Double nested subselects
1.3) Subselects used in the where clause and in the having clause
     simultaneously 1.4) Union Selects using having 1.5) Indexes
on the base relations are used correctly 1.6) Unallowed Queries
are prevented (e.g. qualifications in the
     having clause that belong to the where clause) 1.7) Insert
into as select

2) Queries using the having clause on view relations also work
   but there are some restrictions:

2.1) Create View as Select ... Having ...; using base tables in
the select 2.1.1) The Query rewrite system:

2.1.2) Why are only simple queries allowed against a view from 2.1)
? 2.2) Select ... from testview1, testview2, ... having...; 3) Bug
in ExecMergeJoin ??


Regards Stefan

members to be clearer.  Fix cost computation for these.