diff --git a/doc/TODO b/doc/TODO index 65f6544567b778e56919c9313c37a032d4bd23a8..ffabcb7cf7abc2cf4397caf1be48230288ddedc2 100644 --- a/doc/TODO +++ b/doc/TODO @@ -255,6 +255,7 @@ MISC * Improve statistics storage in pg_class [performance] * Improve VACUUM speed with indexes [vacuum] * -BSD/OS does not support locale because there is no LC_MESSAGES (Bruce) +* Add connection pooling [pool] SOURCE CODE ----------- diff --git a/doc/TODO.detail/pool b/doc/TODO.detail/pool new file mode 100644 index 0000000000000000000000000000000000000000..7d72d9068c5e09e453dc98673383e175814d9732 --- /dev/null +++ b/doc/TODO.detail/pool @@ -0,0 +1,641 @@ +From pgsql-hackers-owner+M4897@hub.org Wed Jul 12 00:15:33 2000 +Received: from hub.org (root@hub.org [216.126.84.1]) + by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id AAA06129 + for ; Wed, 12 Jul 2000 00:15:32 -0400 (EDT) +Received: from hub.org (majordom@localhost [127.0.0.1]) + by hub.org (8.10.1/8.10.1) with SMTP id e6C4FiW14410; + Wed, 12 Jul 2000 00:15:44 -0400 (EDT) +Received: from onyx-technologies.com (iron.onyx-technologies.com [216.205.44.194] (may be forged)) + by hub.org (8.10.1/8.10.1) with ESMTP id e6C4ECW07902 + for ; Wed, 12 Jul 2000 00:14:12 -0400 (EDT) +Received: from onyx-technologies.com (collins.onyx-technologies.com [192.168.188.10]) + by onyx-technologies.com (8.9.2/8.9.0) with ESMTP id AAA14868 + for ; Wed, 12 Jul 2000 00:11:43 -0400 (EDT) +Message-ID: <396BE1B6.F755C5CE@onyx-technologies.com> +Date: Tue, 11 Jul 2000 23:10:46 -0400 +From: Jeffery Collins +X-Mailer: Mozilla 4.7 [en] (X11; I; Linux 2.2.14-15mdk i686) +X-Accept-Language: en +MIME-Version: 1.0 +To: pgsql-hackers@hub.org +Subject: Re: [HACKERS] Connection pooling. +References: <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> +Content-Type: text/plain; charset=us-ascii +Content-Transfer-Encoding: 7bit +X-Mailing-List: pgsql-hackers@postgresql.org +Precedence: bulk +Sender: pgsql-hackers-owner@hub.org +Status: ORr + +It seems like a first step would be to just have postmaster cache unused +connections. In other words if a client closes a connection, postmaster +keeps the connection and the child process around for the next connect +request. This has many of your advantages, but not all. However, it seems +like it would be simpler than attempting to multiplex a connection between +multiple clients. + +Jeff + +> +> Alfred Perlstein wrote: +> > +> > In an effort to complicate the postmaster beyond recognition I'm +> > proposing an idea that I hope can be useful to the developers. +> > +> > Connection pooling: +> > +> > The idea is to have the postmaster multiplex and do hand-offs of +> > database connections to other postgresql processes when the max +> > connections has been exceeded. +> > +> > This allows several gains: +> > +> > 1) Postgresql can support a large number of connections without +> > requiring a large amount of processes to do so. +> > +> > 2) Connection startup/finish will be cheaper because Postgresql +> > processes will not exit and need to reninit things such as shared +> > memory attachments and file opens. This will also reduce the load +> > on the supporting operating system and make postgresql much 'cheaper' +> > to run on systems that don't support the fork() model of execution +> > gracefully. +> > +> > 3) Long running connections can be preempted at transaction boundries +> > allowing other connections to gain process timeslices from the +> > connection pool. +> > +> > The idea is to make the postmaster that accepts connections a broker +> > for the connections. It will dole out descriptors using file +> > descriptor passing to children. If there's a demand for connections +> > meaning that all the postmasters are busy and there are pending +> > connections the postmaster can ask for a yeild on one of the +> > connections. +> > +> > A yeild involves the child postgresql process passing back the +> > client connection at a transaction boundry (between transactions) +> > so it can later be given to another (perhaps the same) child process. +> > +> > I spoke with Bruce briefly about this and he suggested that system +> > tables containing unique IDs could be used to identify passed +> > connections to the children and back to the postmaster. +> > +> > When a handoff occurs, the descriptor along with an ID referencing +> > things like temp tables and enviornment variables and authentication +> > information could be handed out as well allowing the child to resume +> > service to the interrupted connection. +> > +> > I really don't have the knowledge of Postgresql internals to +> > accomplish this, but the concepts are simple and the gains would +> > seem to be very high. +> > +> > Comments? +> > +> > -- +> > -Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] +> > "I have the heart of a child; I keep it in a jar on my desk." + + +From pgsql-hackers-owner+M4904@hub.org Wed Jul 12 01:24:09 2000 +Received: from hub.org (root@hub.org [216.126.84.1]) + by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id BAA06757 + for ; Wed, 12 Jul 2000 01:24:08 -0400 (EDT) +Received: from hub.org (majordom@localhost [127.0.0.1]) + by hub.org (8.10.1/8.10.1) with SMTP id e6C5OLW65679; + Wed, 12 Jul 2000 01:24:21 -0400 (EDT) +Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20]) + by hub.org (8.10.1/8.10.1) with ESMTP id e6C5MkW61040 + for ; Wed, 12 Jul 2000 01:22:46 -0400 (EDT) +Received: (from bright@localhost) + by fw.wintelcom.net (8.10.0/8.10.0) id e6C5Md429901; + Tue, 11 Jul 2000 22:22:39 -0700 (PDT) +Date: Tue, 11 Jul 2000 22:22:39 -0700 +From: Alfred Perlstein +To: Chris Bitmead +Cc: pgsql-hackers@hub.org +Subject: Re: [HACKERS] Connection pooling. +Message-ID: <20000711222239.X25571@fw.wintelcom.net> +References: <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> +Mime-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +User-Agent: Mutt/1.2i +In-Reply-To: <396BEA84.1A06F51F@nimrod.itg.telecom.com.au>; from chrisb@nimrod.itg.telstra.com.au on Wed, Jul 12, 2000 at 01:48:20PM +1000 +X-Mailing-List: pgsql-hackers@postgresql.org +Precedence: bulk +Sender: pgsql-hackers-owner@hub.org +Status: OR + +* Chris Bitmead [000711 20:53] wrote: +> +> Seems a lot trickier than you think. A backend can only be running +> one transaction at a time, so you'd have to keep track of which backends +> are in the middle of a transaction. I can imagine race conditions here. +> And backends can have contexts that are set by various clients using +> SET and friends. Then you'd have to worry about authentication each +> time. And you'd have to have algorithms for cleaning up old processes +> and/or dead processes. It all really sounds a bit hard. + +The backends can simply inform the postmaster when they are ready +either because they are done with a connection or because they +have just closed a transaction. + +All the state (auth/temp tables) can be held in the system tables. + +It's complicated, but no where on the order of something like +a new storage manager. + +-Alfred + +From bright@fw.wintelcom.net Wed Jul 12 01:34:30 2000 +Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20]) + by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id BAA06793 + for ; Wed, 12 Jul 2000 01:34:29 -0400 (EDT) +Received: (from bright@localhost) + by fw.wintelcom.net (8.10.0/8.10.0) id e6C5Z1f00384; + Tue, 11 Jul 2000 22:35:01 -0700 (PDT) +Date: Tue, 11 Jul 2000 22:35:00 -0700 +From: Alfred Perlstein +To: Bruce Momjian +Cc: Jeffery Collins , pgsql-hackers@hub.org +Subject: Re: [HACKERS] Connection pooling. +Message-ID: <20000711223500.Z25571@fw.wintelcom.net> +References: <396BE1B6.F755C5CE@onyx-technologies.com> <200007120428.AAA06357@candle.pha.pa.us> +Mime-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +User-Agent: Mutt/1.2i +In-Reply-To: <200007120428.AAA06357@candle.pha.pa.us>; from pgman@candle.pha.pa.us on Wed, Jul 12, 2000 at 12:28:46AM -0400 +Status: OR + +* Bruce Momjian [000711 21:31] wrote: +> > It seems like a first step would be to just have postmaster cache unused +> > connections. In other words if a client closes a connection, postmaster +> > keeps the connection and the child process around for the next connect +> > request. This has many of your advantages, but not all. However, it seems +> > like it would be simpler than attempting to multiplex a connection between +> > multiple clients. +> > +> +> This does seem like a good optimization. + +I'm not sure if the postmaster is needed besideds just to fork/exec +the backend, if so then when a backend finishes it can just call +accept() on the listening socket inherited from the postmaster to +get the next incomming connection. + +-Alfred + +From pgsql-hackers-owner+M4906@hub.org Wed Jul 12 01:36:44 2000 +Received: from hub.org (root@hub.org [216.126.84.1]) + by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id BAA06806 + for ; Wed, 12 Jul 2000 01:36:44 -0400 (EDT) +Received: from hub.org (majordom@localhost [127.0.0.1]) + by hub.org (8.10.1/8.10.1) with SMTP id e6C5akW94517; + Wed, 12 Jul 2000 01:36:46 -0400 (EDT) +Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20]) + by hub.org (8.10.1/8.10.1) with ESMTP id e6C5ZCW88503 + for ; Wed, 12 Jul 2000 01:35:12 -0400 (EDT) +Received: (from bright@localhost) + by fw.wintelcom.net (8.10.0/8.10.0) id e6C5Z1f00384; + Tue, 11 Jul 2000 22:35:01 -0700 (PDT) +Date: Tue, 11 Jul 2000 22:35:00 -0700 +From: Alfred Perlstein +To: Bruce Momjian +Cc: Jeffery Collins , pgsql-hackers@hub.org +Subject: Re: [HACKERS] Connection pooling. +Message-ID: <20000711223500.Z25571@fw.wintelcom.net> +References: <396BE1B6.F755C5CE@onyx-technologies.com> <200007120428.AAA06357@candle.pha.pa.us> +Mime-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +User-Agent: Mutt/1.2i +In-Reply-To: <200007120428.AAA06357@candle.pha.pa.us>; from pgman@candle.pha.pa.us on Wed, Jul 12, 2000 at 12:28:46AM -0400 +X-Mailing-List: pgsql-hackers@postgresql.org +Precedence: bulk +Sender: pgsql-hackers-owner@hub.org +Status: OR + +* Bruce Momjian [000711 21:31] wrote: +> > It seems like a first step would be to just have postmaster cache unused +> > connections. In other words if a client closes a connection, postmaster +> > keeps the connection and the child process around for the next connect +> > request. This has many of your advantages, but not all. However, it seems +> > like it would be simpler than attempting to multiplex a connection between +> > multiple clients. +> > +> +> This does seem like a good optimization. + +I'm not sure if the postmaster is needed besideds just to fork/exec +the backend, if so then when a backend finishes it can just call +accept() on the listening socket inherited from the postmaster to +get the next incomming connection. + +-Alfred + +From pgsql-hackers-owner+M4907@hub.org Wed Jul 12 01:55:39 2000 +Received: from hub.org (root@hub.org [216.126.84.1]) + by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id BAA06881 + for ; Wed, 12 Jul 2000 01:55:38 -0400 (EDT) +Received: from hub.org (majordom@localhost [127.0.0.1]) + by hub.org (8.10.1/8.10.1) with SMTP id e6C5tnW34576; + Wed, 12 Jul 2000 01:55:49 -0400 (EDT) +Received: from sss2.sss.pgh.pa.us (sss.pgh.pa.us [209.114.166.2]) + by hub.org (8.10.1/8.10.1) with ESMTP id e6C5rfW28119 + for ; Wed, 12 Jul 2000 01:53:42 -0400 (EDT) +Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) + by sss2.sss.pgh.pa.us (8.9.3/8.9.3) with ESMTP id BAA21895; + Wed, 12 Jul 2000 01:52:56 -0400 (EDT) +To: Chris Bitmead +cc: Alfred Perlstein , pgsql-hackers@hub.org +Subject: Re: [HACKERS] Connection pooling. +In-reply-to: <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> +References: <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> +Comments: In-reply-to Chris Bitmead + message dated "Wed, 12 Jul 2000 13:48:20 +1000" +Date: Wed, 12 Jul 2000 01:52:56 -0400 +Message-ID: <21892.963381176@sss.pgh.pa.us> +From: Tom Lane +X-Mailing-List: pgsql-hackers@postgresql.org +Precedence: bulk +Sender: pgsql-hackers-owner@hub.org +Status: OR + +Chris Bitmead writes: +> Seems a lot trickier than you think. A backend can only be running +> one transaction at a time, so you'd have to keep track of which backends +> are in the middle of a transaction. I can imagine race conditions here. + +Aborting out of a transaction is no problem; we have code for that +anyway. More serious problems: + +* We have no code for reassigning a backend to a different database, + so the pooling would have to be per-database. + +* AFAIK there is no portable way to pass a socket connection from the + postmaster to an already-existing backend process. If you do a + fork() then the connection is inherited ... otherwise you've got a + problem. (You could work around this if the postmaster relays + every single byte in both directions between client and backend, + but the performance problems with that should be obvious.) + +> And backends can have contexts that are set by various clients using +> SET and friends. + +Resetting SET variables would be a problem, and there's also the +assigned user name to be reset. This doesn't seem impossible, but +it does seem tedious and error-prone. (OTOH, Peter E's recent work +on guc.c might have unified option-handling enough to bring it +within reason.) + +The killer problem here is that you can't hand off a connection +accepted by the postmaster to a backend except by fork() --- at least +not with methods that work on a wide variety of Unixen. Unless someone +has a way around that, I think the idea is dead in the water; the lesser +issues don't matter. + + regards, tom lane + +From pgsql-hackers-owner+M4910@hub.org Wed Jul 12 02:24:16 2000 +Received: from hub.org (root@hub.org [216.126.84.1]) + by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id CAA11184 + for ; Wed, 12 Jul 2000 02:24:15 -0400 (EDT) +Received: from hub.org (majordom@localhost [127.0.0.1]) + by hub.org (8.10.1/8.10.1) with SMTP id e6C6OAW98187; + Wed, 12 Jul 2000 02:24:10 -0400 (EDT) +Received: from acheron.rime.com.au (root@albatr.lnk.telstra.net [139.130.54.222]) + by hub.org (8.10.1/8.10.1) with ESMTP id e6C6MZW95741 + for ; Wed, 12 Jul 2000 02:22:36 -0400 (EDT) +Received: from oberon (Oberon.rime.com.au [203.8.195.100]) + by acheron.rime.com.au (8.9.3/8.9.3) with SMTP id QAA12845; + Wed, 12 Jul 2000 16:16:23 +1000 +Message-Id: <3.0.5.32.20000712162210.0098fb00@mail.rhyme.com.au> +X-Sender: pjw@mail.rhyme.com.au +X-Mailer: QUALCOMM Windows Eudora Pro Version 3.0.5 (32) +Date: Wed, 12 Jul 2000 16:22:10 +1000 +To: Tom Lane , + Chris Bitmead +From: Philip Warner +Subject: Re: [HACKERS] Connection pooling. +Cc: Alfred Perlstein , pgsql-hackers@hub.org +In-Reply-To: <21892.963381176@sss.pgh.pa.us> +References: <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> + <20000711185318.W25571@fw.wintelcom.net> + <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> +Mime-Version: 1.0 +Content-Type: text/plain; charset="us-ascii" +X-Mailing-List: pgsql-hackers@postgresql.org +Precedence: bulk +Sender: pgsql-hackers-owner@hub.org +Status: OR + +At 01:52 12/07/00 -0400, Tom Lane wrote: +> +>The killer problem here is that you can't hand off a connection +>accepted by the postmaster to a backend except by fork() --- at least +>not with methods that work on a wide variety of Unixen. Unless someone +>has a way around that, I think the idea is dead in the water; the lesser +>issues don't matter. +> + +My understanding of pg client interfaces is that the client uses ont of the +pg interface libraries to make a connection to the db; they specify host & +port and get back some kind of connection object. + +What stops the interface library from using the host & port to talk to the +postmaster, find the host & port the spare db server, then connect directly +to the server? This second connection is passed back in the connection object. + +When the client disconnects from the server, it tells the postmaster it's +available again etc. + +ie. in very rough terms: + + client calls interface to connect + + interface talks to postmaster on port 5432, says "I want a server for +xyz db" + + postmaster replies with "Try port ABCD" OR "no servers available" + postmaster marks the nominated server as 'used' + postmaster disconnects from client + + interface connects to port ABCD as per normal protocols + interface fills in connection object & returns + + ...client does some work... + + client disconnects + + db server tells postmaster it's available again. + + +There would also need to be timeout code to handle the case where the +interface did not do the second connect. + +You could also have the interface allocate a port send it's number to the +postmaster then listen on it, but I think that would represent a potential +security hole. + + +---------------------------------------------------------------- +Philip Warner | __---_____ +Albatross Consulting Pty. Ltd. |----/ - \ +(A.C.N. 008 659 498) | /(@) ______---_ +Tel: (+61) 0500 83 82 81 | _________ \ +Fax: (+61) 0500 83 82 82 | ___________ | +Http://www.rhyme.com.au | / \| + | --________-- +PGP key available upon request, | / +and from pgp5.ai.mit.edu:11371 |/ + +From pgsql-hackers-owner+M4912@hub.org Wed Jul 12 02:32:21 2000 +Received: from hub.org (root@hub.org [216.126.84.1]) + by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id CAA11228 + for ; Wed, 12 Jul 2000 02:32:20 -0400 (EDT) +Received: from hub.org (majordom@localhost [127.0.0.1]) + by hub.org (8.10.1/8.10.1) with SMTP id e6C6WWW18412; + Wed, 12 Jul 2000 02:32:32 -0400 (EDT) +Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20]) + by hub.org (8.10.1/8.10.1) with ESMTP id e6C6UwW16062 + for ; Wed, 12 Jul 2000 02:30:58 -0400 (EDT) +Received: (from bright@localhost) + by fw.wintelcom.net (8.10.0/8.10.0) id e6C6Uov01852; + Tue, 11 Jul 2000 23:30:50 -0700 (PDT) +Date: Tue, 11 Jul 2000 23:30:49 -0700 +From: Alfred Perlstein +To: Tom Lane +Cc: Chris Bitmead , pgsql-hackers@hub.org +Subject: Re: [HACKERS] Connection pooling. +Message-ID: <20000711233049.A25571@fw.wintelcom.net> +References: <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> <21892.963381176@sss.pgh.pa.us> +Mime-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +User-Agent: Mutt/1.2i +In-Reply-To: <21892.963381176@sss.pgh.pa.us>; from tgl@sss.pgh.pa.us on Wed, Jul 12, 2000 at 01:52:56AM -0400 +X-Mailing-List: pgsql-hackers@postgresql.org +Precedence: bulk +Sender: pgsql-hackers-owner@hub.org +Status: OR + +* Tom Lane [000711 22:53] wrote: +> Chris Bitmead writes: +> > Seems a lot trickier than you think. A backend can only be running +> > one transaction at a time, so you'd have to keep track of which backends +> > are in the middle of a transaction. I can imagine race conditions here. +> +> Aborting out of a transaction is no problem; we have code for that +> anyway. More serious problems: +> +> * We have no code for reassigning a backend to a different database, +> so the pooling would have to be per-database. + +That would need to be fixed. How difficult would that be? + +> * AFAIK there is no portable way to pass a socket connection from the +> postmaster to an already-existing backend process. If you do a +> fork() then the connection is inherited ... otherwise you've got a +> problem. (You could work around this if the postmaster relays +> every single byte in both directions between client and backend, +> but the performance problems with that should be obvious.) + +no, see below. + +> > And backends can have contexts that are set by various clients using +> > SET and friends. +> +> Resetting SET variables would be a problem, and there's also the +> assigned user name to be reset. This doesn't seem impossible, but +> it does seem tedious and error-prone. (OTOH, Peter E's recent work +> on guc.c might have unified option-handling enough to bring it +> within reason.) + +What can be done is that each incomming connection can be assigned an +ID into a system table. As options are added the system would assign +them to key-value pairs in this table. Once someone detects that the +remote side has closed the connection the data can be destroyed, but +until then along with the descriptor passing the ID of the client +as an index into the table can be passed for the backend to fetch. + +> The killer problem here is that you can't hand off a connection +> accepted by the postmaster to a backend except by fork() --- at least +> not with methods that work on a wide variety of Unixen. Unless someone +> has a way around that, I think the idea is dead in the water; the lesser +> issues don't matter. + +The code has been around since 4.2BSD, it takes a bit of #ifdef to +get it right on all systems but it's not impossible, have a look at +http://www.fhttpd.org/ for a web server that does this in a portable +fashion. + +I should have a library whipped up for you guys really soon now +to handle the descriptor and message passing. + +-- +-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] +"I have the heart of a child; I keep it in a jar on my desk." + +From pgsql-hackers-owner+M4913@hub.org Wed Jul 12 03:06:54 2000 +Received: from hub.org (root@hub.org [216.126.84.1]) + by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id DAA11529 + for ; Wed, 12 Jul 2000 03:06:53 -0400 (EDT) +Received: from hub.org (majordom@localhost [127.0.0.1]) + by hub.org (8.10.1/8.10.1) with SMTP id e6C76ZW95615; + Wed, 12 Jul 2000 03:06:35 -0400 (EDT) +Received: from sss2.sss.pgh.pa.us (sss.pgh.pa.us [209.114.166.2]) + by hub.org (8.10.1/8.10.1) with ESMTP id e6C74gW93358 + for ; Wed, 12 Jul 2000 03:04:42 -0400 (EDT) +Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) + by sss2.sss.pgh.pa.us (8.9.3/8.9.3) with ESMTP id DAA22136; + Wed, 12 Jul 2000 03:04:13 -0400 (EDT) +To: Alfred Perlstein +cc: Chris Bitmead , pgsql-hackers@hub.org +Subject: Re: [HACKERS] Connection pooling. +In-reply-to: <20000711233049.A25571@fw.wintelcom.net> +References: <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> <21892.963381176@sss.pgh.pa.us> <20000711233049.A25571@fw.wintelcom.net> +Comments: In-reply-to Alfred Perlstein + message dated "Tue, 11 Jul 2000 23:30:49 -0700" +Date: Wed, 12 Jul 2000 03:04:13 -0400 +Message-ID: <22133.963385453@sss.pgh.pa.us> +From: Tom Lane +X-Mailing-List: pgsql-hackers@postgresql.org +Precedence: bulk +Sender: pgsql-hackers-owner@hub.org +Status: OR + +Alfred Perlstein writes: +> * Tom Lane [000711 22:53] wrote: +>> The killer problem here is that you can't hand off a connection +>> accepted by the postmaster to a backend except by fork() --- at least +>> not with methods that work on a wide variety of Unixen. + +> The code has been around since 4.2BSD, it takes a bit of #ifdef to +> get it right on all systems but it's not impossible, have a look at +> http://www.fhttpd.org/ for a web server that does this in a portable +> fashion. + +I looked at this to see if it would teach me something I didn't know. +It doesn't. It depends on sendmsg() which is a BSD-ism and not very +portable. + + regards, tom lane + +From pgsql-hackers-owner+M4914@hub.org Wed Jul 12 03:12:40 2000 +Received: from hub.org (root@hub.org [216.126.84.1]) + by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id DAA11597 + for ; Wed, 12 Jul 2000 03:12:39 -0400 (EDT) +Received: from hub.org (majordom@localhost [127.0.0.1]) + by hub.org (8.10.1/8.10.1) with SMTP id e6C7CjW13459; + Wed, 12 Jul 2000 03:12:45 -0400 (EDT) +Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20]) + by hub.org (8.10.1/8.10.1) with ESMTP id e6C7B8W07036 + for ; Wed, 12 Jul 2000 03:11:08 -0400 (EDT) +Received: (from bright@localhost) + by fw.wintelcom.net (8.10.0/8.10.0) id e6C79lE02841; + Wed, 12 Jul 2000 00:09:47 -0700 (PDT) +Date: Wed, 12 Jul 2000 00:09:47 -0700 +From: Alfred Perlstein +To: Tom Lane +Cc: Chris Bitmead , pgsql-hackers@hub.org +Subject: Re: [HACKERS] Connection pooling. +Message-ID: <20000712000947.D25571@fw.wintelcom.net> +References: <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> <21892.963381176@sss.pgh.pa.us> <20000711233049.A25571@fw.wintelcom.net> <22133.963385453@sss.pgh.pa.us> +Mime-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +User-Agent: Mutt/1.2i +In-Reply-To: <22133.963385453@sss.pgh.pa.us>; from tgl@sss.pgh.pa.us on Wed, Jul 12, 2000 at 03:04:13AM -0400 +X-Mailing-List: pgsql-hackers@postgresql.org +Precedence: bulk +Sender: pgsql-hackers-owner@hub.org +Status: OR + +* Tom Lane [000712 00:04] wrote: +> Alfred Perlstein writes: +> > * Tom Lane [000711 22:53] wrote: +> >> The killer problem here is that you can't hand off a connection +> >> accepted by the postmaster to a backend except by fork() --- at least +> >> not with methods that work on a wide variety of Unixen. +> +> > The code has been around since 4.2BSD, it takes a bit of #ifdef to +> > get it right on all systems but it's not impossible, have a look at +> > http://www.fhttpd.org/ for a web server that does this in a portable +> > fashion. +> +> I looked at this to see if it would teach me something I didn't know. +> It doesn't. It depends on sendmsg() which is a BSD-ism and not very +> portable. + +It's also specified by Posix.1g if that means anything. + +-Alfred + +From pgsql-hackers-owner+M4916@hub.org Wed Jul 12 03:49:58 2000 +Received: from hub.org (root@hub.org [216.126.84.1]) + by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id DAA11736 + for ; Wed, 12 Jul 2000 03:49:58 -0400 (EDT) +Received: from hub.org (majordom@localhost [127.0.0.1]) + by hub.org (8.10.1/8.10.1) with SMTP id e6C7oBW95547; + Wed, 12 Jul 2000 03:50:11 -0400 (EDT) +Received: from sss2.sss.pgh.pa.us (sss.pgh.pa.us [209.114.166.2]) + by hub.org (8.10.1/8.10.1) with ESMTP id e6C7mPW93141 + for ; Wed, 12 Jul 2000 03:48:25 -0400 (EDT) +Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) + by sss2.sss.pgh.pa.us (8.9.3/8.9.3) with ESMTP id DAA22297; + Wed, 12 Jul 2000 03:47:37 -0400 (EDT) +To: Philip Warner +cc: Chris Bitmead , + Alfred Perlstein , pgsql-hackers@hub.org +Subject: Re: [HACKERS] Connection pooling. +In-reply-to: <3.0.5.32.20000712162210.0098fb00@mail.rhyme.com.au> +References: <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> <3.0.5.32.20000712162210.0098fb00@mail.rhyme.com.au> +Comments: In-reply-to Philip Warner + message dated "Wed, 12 Jul 2000 16:22:10 +1000" +Date: Wed, 12 Jul 2000 03:47:37 -0400 +Message-ID: <22294.963388057@sss.pgh.pa.us> +From: Tom Lane +X-Mailing-List: pgsql-hackers@postgresql.org +Precedence: bulk +Sender: pgsql-hackers-owner@hub.org +Status: OR + +Philip Warner writes: +> What stops the interface library from using the host & port to talk to +> the postmaster, find the host & port the spare db server, then connect +> directly to the server? + +You're assuming that we can change the on-the-wire protocol freely and +only the API presented by the client libraries matters. In a perfect +world that might be true, but reality is that we can't change the wire +protocol easily. If we do, it breaks all existing precompiled clients. +Updating clients can be an extremely painful experience in multiple- +machine installations. + +Also, we don't have just one set of client libraries to fix. There are +at least three client-side implementations that don't depend on libpq. + +We have done protocol updates in the past --- in fact I was responsible +for the last one. (And I'm still carrying the scars, which is why I'm +not too enthusiastic about another one.) It's not impossible, but it +needs more evidence than "this should speed up connections by +I-don't-know-how-much"... + +It might also be worth pointing out that the goal was to speed up the +end-to-end connection time. Redirecting as you suggest is not free +(at minimum it would appear to require two TCP connection setups and two +authentication cycles). What evidence have you got that it'd be faster +than spawning a new backend? + +I tend to agree with the opinion that connection-pooling on the client +side offers more bang for the buck than we could hope to get by doing +surgery on the postmaster/backend setup. + +Also, to return to the original point, AFAIK we have not tried hard +to cut the backend startup time, other than the work that was done +a year or so back to eliminate exec() of a separate executable. +It'd be worth looking to see what could be done there with zero +impact on existing clients. + + regards, tom lane +