diff --git a/doc/TODO.detail/crossdb b/doc/TODO.detail/crossdb deleted file mode 100644 index a68fb19828..0000000000 --- a/doc/TODO.detail/crossdb +++ /dev/null @@ -1,771 +0,0 @@ -From pgsql-hackers-owner+M16329=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 13:31:28 2001 -Return-path: -Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6IVRZ13376 - for ; Thu, 6 Dec 2001 13:31:27 -0500 (EST) -Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6IVQa26949 - for ; Thu, 6 Dec 2001 13:31:26 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6IRvR61959 - for ; Thu, 6 Dec 2001 12:29:06 -0600 (CST) - (envelope-from pgsql-hackers-owner+M16329=candle.pha.pa.us=pgman@postgresql.org) -Received: from gromit.dotclick.com (ipn9-f8366.net-resource.net [216.204.83.66]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6IQ2m78192 - for ; Thu, 6 Dec 2001 13:26:05 -0500 (EST) - (envelope-from markw@mohawksoft.com) -Received: from mohawksoft.com (IDENT:markw@localhost.localdomain [127.0.0.1]) - by gromit.dotclick.com (8.9.3/8.9.3) with ESMTP id NAA10076 - for ; Thu, 6 Dec 2001 13:28:04 -0500 -Message-ID: <3C0FB8B4.382C7736@mohawksoft.com> -Date: Thu, 06 Dec 2001 13:28:04 -0500 -From: mlw -X-Mailer: Mozilla 4.75 [en] (X11; U; Linux 2.4.16 i686) -X-Accept-Language: en -MIME-Version: 1.0 -To: "pgsql-hackers@postgresql.org" -Subject: [HACKERS] Remote connections? -Content-Type: text/plain; charset=us-ascii -Content-Transfer-Encoding: 7bit -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -I just found out something about Oracle which that looks like something -that could be doable in PostgreSQL. - -What do you all think: - -Oracle's version is something like this: - -create [public] database link using [...] - -select * from sometable@remotelink - - -I was thinking how this could be done with postgreSQL. How hard would it -be to make something that is similar to a view, but executes a query -remotely? I envision something like this: - -create [public] link name query using [...] - -The table link will be similar to a view. It could be used like this: - -CREATE LINK test as select * from test WITH 'user=postgres host=remote -db=data'; - -SELECT * from test; - -or - -SELECT * from fubar join test on (fubar.id = test.id) ; - -So, what do you think? Impossible, possible, too hard? too easy? - ----------------------------(end of broadcast)--------------------------- -TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org - -From pgsql-hackers-owner+M16331=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 15:12:28 2001 -Return-path: -Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6KCQZ19987 - for ; Thu, 6 Dec 2001 15:12:27 -0500 (EST) -Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6KCQa13967 - for ; Thu, 6 Dec 2001 15:12:26 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6K6nR65020 - for ; Thu, 6 Dec 2001 14:07:54 -0600 (CST) - (envelope-from pgsql-hackers-owner+M16331=candle.pha.pa.us=pgman@postgresql.org) -Received: from ece.rice.edu (ece.rice.edu [128.42.4.34]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6K6Im96910 - for ; Thu, 6 Dec 2001 15:06:18 -0500 (EST) - (envelope-from reedstrm@rice.edu) -Received: from localhost (localhost [127.0.0.1]) - by ece.rice.edu (Postfix) with ESMTP - id A9E4E68A1D; Thu, 6 Dec 2001 14:06:17 -0600 (CST) -Received: from wallace.ece.rice.edu (wallace.ece.rice.edu [128.42.12.154]) - by ece.rice.edu (Postfix) with ESMTP - id EA06668A17; Thu, 6 Dec 2001 14:06:16 -0600 (CST) -Received: from reedstrm by wallace.ece.rice.edu with local (Exim 3.22 #1 (Debian)) - id 16C4me-0002uX-00; Thu, 06 Dec 2001 14:06:16 -0600 -Date: Thu, 6 Dec 2001 14:06:16 -0600 -From: "Ross J. Reedstrom" -To: mlw -cc: "pgsql-hackers@postgresql.org" -Subject: Re: [HACKERS] Remote connections? -Message-ID: <20011206140616.C10995@rice.edu> -Mail-Followup-To: "Ross J. Reedstrom" , - mlw , - "pgsql-hackers@postgresql.org" -References: <3C0FB8B4.382C7736@mohawksoft.com> -MIME-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -In-Reply-To: <3C0FB8B4.382C7736@mohawksoft.com> -User-Agent: Mutt/1.3.18i -X-Virus-Scanned: by AMaViS snapshot-20010714 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -On Thu, Dec 06, 2001 at 01:28:04PM -0500, mlw wrote: -> I just found out something about Oracle which that looks like something -> that could be doable in PostgreSQL. -> -> What do you all think: -> -> Oracle's version is something like this: -> -> create [public] database link using [...] -> -> select * from sometable@remotelink -> -> -> I was thinking how this could be done with postgreSQL. How hard would it -> be to make something that is similar to a view, but executes a query -> remotely? I envision something like this: -> -> create [public] link name query using [...] -> -> The table link will be similar to a view. It could be used like this: -> -> CREATE LINK test as select * from test WITH 'user=postgres host=remote -> db=data'; -> -> SELECT * from test; -> -> or -> -> SELECT * from fubar join test on (fubar.id = test.id) ; -> -> So, what do you think? Impossible, possible, too hard? too easy? - -Here we come, full circle. This is just about where I came on board. -Many moons ago, I started looking at Mariposa, in the hopes of forward -patching it into PostgreSQL, and generalizing the 'remote' part to allow -exactly the sort of access you described above. - -The biggest problem with this is transactional semantics: you need -two-stage commits to get this right, and we don't hav'em. (Has there -been an indepth discussion concerning what how hard it would be to do -that with postgresql?) - -The _actual_ biggest problem was my lack of knowledge of the PostgreSQL -codebase ;-) - -Ross --- -Ross Reedstrom, Ph.D. reedstrm@rice.edu -Executive Director phone: 713-348-6166 -Gulf Coast Consortium for Bioinformatics fax: 713-348-6182 -Rice University MS-39 -Houston, TX 77005 - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M16332=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 15:31:27 2001 -Return-path: -Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6KVQZ21158 - for ; Thu, 6 Dec 2001 15:31:26 -0500 (EST) -Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6KVQa22900 - for ; Thu, 6 Dec 2001 15:31:26 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6KRrR65596 - for ; Thu, 6 Dec 2001 14:28:55 -0600 (CST) - (envelope-from pgsql-hackers-owner+M16332=candle.pha.pa.us=pgman@postgresql.org) -Received: from gromit.dotclick.com (ipn9-f8366.net-resource.net [216.204.83.66]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6KJXm97564 - for ; Thu, 6 Dec 2001 15:19:33 -0500 (EST) - (envelope-from markw@mohawksoft.com) -Received: from mohawksoft.com (IDENT:markw@localhost.localdomain [127.0.0.1]) - by gromit.dotclick.com (8.9.3/8.9.3) with ESMTP id PAA10771; - Thu, 6 Dec 2001 15:21:13 -0500 -Message-ID: <3C0FD339.6F663329@mohawksoft.com> -Date: Thu, 06 Dec 2001 15:21:13 -0500 -From: mlw -X-Mailer: Mozilla 4.75 [en] (X11; U; Linux 2.4.16 i686) -X-Accept-Language: en -MIME-Version: 1.0 -To: "Ross J. Reedstrom" -cc: "pgsql-hackers@postgresql.org" -Subject: Re: [HACKERS] Remote connections? -References: <3C0FB8B4.382C7736@mohawksoft.com> <20011206140616.C10995@rice.edu> -Content-Type: text/plain; charset=us-ascii -Content-Transfer-Encoding: 7bit -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -"Ross J. Reedstrom" wrote: -> -> On Thu, Dec 06, 2001 at 01:28:04PM -0500, mlw wrote: -> > I just found out something about Oracle which that looks like something -> > that could be doable in PostgreSQL. -> > -> > What do you all think: -> > -> > Oracle's version is something like this: -> > -> > create [public] database link using [...] -> > -> > select * from sometable@remotelink -> > -> > -> > I was thinking how this could be done with postgreSQL. How hard would it -> > be to make something that is similar to a view, but executes a query -> > remotely? I envision something like this: -> > -> > create [public] link name query using [...] -> > -> > The table link will be similar to a view. It could be used like this: -> > -> > CREATE LINK test as select * from test WITH 'user=postgres host=remote -> > db=data'; -> > -> > SELECT * from test; -> > -> > or -> > -> > SELECT * from fubar join test on (fubar.id = test.id) ; -> > -> > So, what do you think? Impossible, possible, too hard? too easy? -> -> Here we come, full circle. This is just about where I came on board. -> Many moons ago, I started looking at Mariposa, in the hopes of forward -> patching it into PostgreSQL, and generalizing the 'remote' part to allow -> exactly the sort of access you described above. -> -> The biggest problem with this is transactional semantics: you need -> two-stage commits to get this right, and we don't hav'em. (Has there -> been an indepth discussion concerning what how hard it would be to do -> that with postgresql?) -> -> The _actual_ biggest problem was my lack of knowledge of the PostgreSQL -> codebase ;-) - -I think we can we can dispense worrying about two stage commits, if we -assume that remote connections are treated as views with no rules. As -long as remote tables are "read only" then the implementation is much -easier. - -I too find the internals of PostgreSQL virtually incomprehensible at the -internal level. If there were a document somewhere which published how a -function could return multiple tuples, remote views would be a trivial -undertaking. It could look like: - -select * from remote('select *from table', 'user=postgres host=outland -db=remote'); - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M16335=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 17:11:29 2001 -Return-path: -Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6MBSZ06676 - for ; Thu, 6 Dec 2001 17:11:28 -0500 (EST) -Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6MBSq07059 - for ; Thu, 6 Dec 2001 17:11:28 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6M6sR68489 - for ; Thu, 6 Dec 2001 16:08:16 -0600 (CST) - (envelope-from pgsql-hackers-owner+M16335=candle.pha.pa.us=pgman@postgresql.org) -Received: from mx1.relaypoint.net (ns2.generalbroadband.com [64.32.62.5]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6M3Im04451 - for ; Thu, 6 Dec 2001 17:03:18 -0500 (EST) - (envelope-from joseph.conway@home.com) -Received: from [206.19.64.3] (account jconway@wwc.com HELO home.com) - by mx1.relaypoint.net (CommuniGate Pro SMTP 3.4.8) - with ESMTP id 1707032; Thu, 06 Dec 2001 14:03:14 -0800 -Message-ID: <3C0FEB2C.70000@home.com> -Date: Thu, 06 Dec 2001 14:03:24 -0800 -From: Joe Conway -User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.6+) Gecko/20011126 -X-Accept-Language: en-us -MIME-Version: 1.0 -To: mlw -cc: "Ross J. Reedstrom" , - "pgsql-hackers@postgresql.org" -Subject: Re: [HACKERS] Remote connections? -References: <3C0FB8B4.382C7736@mohawksoft.com> <20011206140616.C10995@rice.edu> <3C0FD339.6F663329@mohawksoft.com> -Content-Type: text/plain; charset=us-ascii; format=flowed -Content-Transfer-Encoding: 7bit -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -mlw wrote: - -> I too find the internals of PostgreSQL virtually incomprehensible at the -> internal level. If there were a document somewhere which published how a -> function could return multiple tuples, remote views would be a trivial -> undertaking. It could look like: -> -> select * from remote('select *from table', 'user=postgres host=outland -> db=remote'); -> - -See contrib/dblink in the 7.2 beta. It was my attempt inspired by -Oracle's dblink and some code that you (mlw) posted a while back. Based -on the limitations wrt returning muliple tuples, I wound up with -something like: - -lt_lcat=# select dblink_tok(t1.dblink_p,0) as f1 from (select -dblink('hostaddr=127.0.0.1 dbname=template1 user=postgres -password=postgres','select proname from pg_proc') as dblink_p) as t1; - -Which, as has been pointed out more than once, is pretty ugly, but at -least it's a start ;-) - - -Joe - - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M16336=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 18:41:31 2001 -Return-path: -Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6NfPZ12249 - for ; Thu, 6 Dec 2001 18:41:25 -0500 (EST) -Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6NfQq03244 - for ; Thu, 6 Dec 2001 18:41:26 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6NbOR70960 - for ; Thu, 6 Dec 2001 17:38:19 -0600 (CST) - (envelope-from pgsql-hackers-owner+M16336=candle.pha.pa.us=pgman@postgresql.org) -Received: from spider.pilosoft.com (p55-222.acedsl.com [160.79.55.222]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6NIgm07232 - for ; Thu, 6 Dec 2001 18:18:43 -0500 (EST) - (envelope-from alex@pilosoft.com) -Received: from localhost (alexmail@localhost) - by spider.pilosoft.com (8.9.3/8.9.3) with ESMTP id SAA23999; - Thu, 6 Dec 2001 18:23:22 -0500 (EST) -Date: Thu, 6 Dec 2001 18:23:22 -0500 (EST) -From: Alex Pilosov -To: mlw -cc: "pgsql-hackers@postgresql.org" -Subject: Re: [HACKERS] Remote connections? -In-Reply-To: <3C0FD339.6F663329@mohawksoft.com> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -On Thu, 6 Dec 2001, mlw wrote: - -> I too find the internals of PostgreSQL virtually incomprehensible at the -> internal level. If there were a document somewhere which published how a -> function could return multiple tuples, remote views would be a trivial -> undertaking. It could look like: -> -> select * from remote('select *from table', 'user=postgres host=outland -> db=remote'); -This isn't possible yet. I was working on implementation of this, about -80% done, but never finished. Now I'm out of time to work more on this for -a while. :( - -Let me know if you want my code. - --alex - - ----------------------------(end of broadcast)--------------------------- -TIP 6: Have you searched our list archives? - -http://archives.postgresql.org - -From pgsql-hackers-owner+M16340=candle.pha.pa.us=pgman@postgresql.org Fri Dec 7 00:32:59 2001 -Return-path: -Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB75WsZ06911 - for ; Fri, 7 Dec 2001 00:32:54 -0500 (EST) -Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB75Wtq16801 - for ; Fri, 7 Dec 2001 00:32:55 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB75SCR80525 - for ; Thu, 6 Dec 2001 23:29:17 -0600 (CST) - (envelope-from pgsql-hackers-owner+M16340=candle.pha.pa.us=pgman@postgresql.org) -Received: from snoopy.mohawksoft.com (h0050bf7a618d.ne.mediaone.net [24.218.67.153]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id fB7593m27913 - for ; Fri, 7 Dec 2001 00:09:03 -0500 (EST) - (envelope-from markw@mohawksoft.com) -Received: from mohawksoft.com (localhost [127.0.0.1]) - by snoopy.mohawksoft.com (8.11.6/8.11.6) with ESMTP id fB7561030613; - Fri, 7 Dec 2001 00:06:01 -0500 -Message-ID: <3C104E38.DA19C867@mohawksoft.com> -Date: Fri, 07 Dec 2001 00:06:01 -0500 -From: mlw -X-Mailer: Mozilla 4.78 [en] (X11; U; Linux 2.4.14ext3 i686) -X-Accept-Language: en -MIME-Version: 1.0 -To: Joe Conway -cc: "Ross J. Reedstrom" , - "pgsql-hackers@postgresql.org" -Subject: Re: [HACKERS] Remote connections? -References: <3C0FB8B4.382C7736@mohawksoft.com> <20011206140616.C10995@rice.edu> <3C0FD339.6F663329@mohawksoft.com> <3C0FEB2C.70000@home.com> -Content-Type: text/plain; charset=us-ascii -Content-Transfer-Encoding: 7bit -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -Hey this looks really cool. It looks like something I was thinking about doing. -I have a couple suggestions that could make it a little better, I hope you will -not be offended. (If you want my help, I'll chip in!) - -Why not use a binary cursor? That way native types can slip through without the -overhead of conversion. - -Right now you get all rows up front, you may be able to increase overall -performance by fetching only a few rows at a time, rather than get everything -all at once. (Think on the order of 4 million rows from your remote query!) -Execute the commit at the end of processing. There are even some asynchronous -functions you may be able to utilize to reduce the I/O bottleneck. Use the -synchronous function first, then before you return initiate an asynchronous -read. Every successive pass through the function, read the newly arrived tuple, -and initiate the next asynchronous read. (The two machine could be processing -the query simultaneously, and this could even IMPROVE performance over a single -system for heavy duty queries.) - -Setup a hash table for field names, rather than requiring field numbers. (Keep -field number for efficiency, of course.) - -You could eliminate having to pass the result pointer around by keeping a -static array in your extension. Use something like Oracle's "contains" notation -of result number. Where each instantiation of "contains()" and "score()" -require an id. i.e. 1,2,3,40 etc. Then hash those numbers into an array. I have -some code that does this for a PostgreSQL extension (it implements contains) on -my website (pgcontains, under download). It is ugly but works for the most -part. - -Seriously, your stuff looks great. I think it could be the beginning of a -fairly usable system for my company. Any help you need/want, just let me know. - - -Joe Conway wrote: -> -> mlw wrote: -> -> > I too find the internals of PostgreSQL virtually incomprehensible at the -> > internal level. If there were a document somewhere which published how a -> > function could return multiple tuples, remote views would be a trivial -> > undertaking. It could look like: -> > -> > select * from remote('select *from table', 'user=postgres host=outland -> > db=remote'); -> > -> -> See contrib/dblink in the 7.2 beta. It was my attempt inspired by -> Oracle's dblink and some code that you (mlw) posted a while back. Based -> on the limitations wrt returning muliple tuples, I wound up with -> something like: -> -> lt_lcat=# select dblink_tok(t1.dblink_p,0) as f1 from (select -> dblink('hostaddr=127.0.0.1 dbname=template1 user=postgres -> password=postgres','select proname from pg_proc') as dblink_p) as t1; -> -> Which, as has been pointed out more than once, is pretty ugly, but at -> least it's a start ;-) -> -> Joe -> -> ---------------------------(end of broadcast)--------------------------- -> TIP 4: Don't 'kill -9' the postmaster - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate -subscribe-nomail command to majordomo@postgresql.org so that your -message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M16344=candle.pha.pa.us=pgman@postgresql.org Fri Dec 7 02:51:51 2001 -Return-path: -Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB77poZ14221 - for ; Fri, 7 Dec 2001 02:51:50 -0500 (EST) -Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB77pqq08152 - for ; Fri, 7 Dec 2001 02:51:52 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB77lwR84105 - for ; Fri, 7 Dec 2001 01:49:04 -0600 (CST) - (envelope-from pgsql-hackers-owner+M16344=candle.pha.pa.us=pgman@postgresql.org) -Received: from ra.sai.msu.su (ra.sai.msu.su [158.250.29.2]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id fB77bmm32783 - for ; Fri, 7 Dec 2001 02:37:49 -0500 (EST) - (envelope-from oleg@sai.msu.su) -Received: from ra (ra [158.250.29.2]) - by ra.sai.msu.su (8.9.3/8.9.3) with ESMTP id KAA04160; - Fri, 7 Dec 2001 10:37:04 +0300 (GMT) -Date: Fri, 7 Dec 2001 10:37:04 +0300 (GMT) -From: Oleg Bartunov -X-X-Sender: -To: mlw -cc: Joe Conway , - "Ross J. Reedstrom" , - "pgsql-hackers@postgresql.org" -Subject: Re: [HACKERS] Remote connections? -In-Reply-To: <3C104E38.DA19C867@mohawksoft.com> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -On Fri, 7 Dec 2001, mlw wrote: - -> -> You could eliminate having to pass the result pointer around by keeping a -> static array in your extension. Use something like Oracle's "contains" notation -> of result number. Where each instantiation of "contains()" and "score()" -> require an id. i.e. 1,2,3,40 etc. Then hash those numbers into an array. I have -> some code that does this for a PostgreSQL extension (it implements contains) on -> my website (pgcontains, under download). It is ugly but works for the most -> part. - -contrib/intarray does this job very well - -> -> Seriously, your stuff looks great. I think it could be the beginning of a -> fairly usable system for my company. Any help you need/want, just let me know. -> -> -> Joe Conway wrote: -> > -> > mlw wrote: -> > -> > > I too find the internals of PostgreSQL virtually incomprehensible at the -> > > internal level. If there were a document somewhere which published how a -> > > function could return multiple tuples, remote views would be a trivial -> > > undertaking. It could look like: -> > > -> > > select * from remote('select *from table', 'user=postgres host=outland -> > > db=remote'); -> > > -> > -> > See contrib/dblink in the 7.2 beta. It was my attempt inspired by -> > Oracle's dblink and some code that you (mlw) posted a while back. Based -> > on the limitations wrt returning muliple tuples, I wound up with -> > something like: -> > -> > lt_lcat=# select dblink_tok(t1.dblink_p,0) as f1 from (select -> > dblink('hostaddr=127.0.0.1 dbname=template1 user=postgres -> > password=postgres','select proname from pg_proc') as dblink_p) as t1; -> > -> > Which, as has been pointed out more than once, is pretty ugly, but at -> > least it's a start ;-) -> > -> > Joe -> > -> > ---------------------------(end of broadcast)--------------------------- -> > TIP 4: Don't 'kill -9' the postmaster -> -> ---------------------------(end of broadcast)--------------------------- -> TIP 3: if posting/reading through Usenet, please send an appropriate -> subscribe-nomail command to majordomo@postgresql.org so that your -> message can get through to the mailing list cleanly -> - - Regards, - Oleg -_____________________________________________________________ -Oleg Bartunov, sci.researcher, hostmaster of AstroNet, -Sternberg Astronomical Institute, Moscow University (Russia) -Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ -phone: +007(095)939-16-83, +007(095)939-23-83 - - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate -subscribe-nomail command to majordomo@postgresql.org so that your -message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M16412=candle.pha.pa.us=pgman@postgresql.org Mon Dec 10 12:35:01 2001 -Return-path: -Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBAHZ0Z09772 - for ; Mon, 10 Dec 2001 12:35:00 -0500 (EST) -Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fBAHZ0a11739 - for ; Mon, 10 Dec 2001 12:35:00 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fBAHRAR20284 - for ; Mon, 10 Dec 2001 11:32:00 -0600 (CST) - (envelope-from pgsql-hackers-owner+M16412=candle.pha.pa.us=pgman@postgresql.org) -Received: from snoopy.mohawksoft.com (h0050bf7a618d.ne.mediaone.net [24.218.67.153]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id fB7DhOm75114 - for ; Fri, 7 Dec 2001 08:43:24 -0500 (EST) - (envelope-from markw@mohawksoft.com) -Received: from mohawksoft.com (localhost [127.0.0.1]) - by snoopy.mohawksoft.com (8.11.6/8.11.6) with ESMTP id fB7De0000437; - Fri, 7 Dec 2001 08:40:01 -0500 -Message-ID: <3C10C6B0.865669C1@mohawksoft.com> -Date: Fri, 07 Dec 2001 08:40:00 -0500 -From: mlw -X-Mailer: Mozilla 4.78 [en] (X11; U; Linux 2.4.14ext3 i686) -X-Accept-Language: en -MIME-Version: 1.0 -To: Oleg Bartunov -cc: Joe Conway , - "Ross J. Reedstrom" , - "pgsql-hackers@postgresql.org" -Subject: Re: [HACKERS] Remote connections? -References: -Content-Type: text/plain; charset=us-ascii -Content-Transfer-Encoding: 7bit -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -The dblink code is a very cool idea. - -It got me thinking, what if, just thinking out load here, it was redesigned as -something a little more grandeous. - -Imagine this: - - -select dblink('select * from table', 'table_name', 'db=oracle.test user=chris -passwd=knight', 1) as t1, dblink('table2_name', 1) as t2 - - -Just something to think about. - -The first instance of dblink would take 4 parameters: query, table which it -returns, connect string, and link token. - -The second instance of dblink would just take the name of the table which it -returns and a link token. - -The cool bit is the notion that the query string could specify different -databases or even .DBF libraries. - -Just something to think about. - -It would REALLY be great if functions could return multiple tuples! - ----------------------------(end of broadcast)--------------------------- -TIP 5: Have you checked our extensive FAQ? - -http://www.postgresql.org/users-lounge/docs/faq.html - -From pgsql-hackers-owner+M16365=candle.pha.pa.us=pgman@postgresql.org Fri Dec 7 12:32:26 2001 -Return-path: -Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB7HWMZ26245 - for ; Fri, 7 Dec 2001 12:32:22 -0500 (EST) -Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB7HWLB14472 - for ; Fri, 7 Dec 2001 12:32:21 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB7HSHR01506 - for ; Fri, 7 Dec 2001 11:29:07 -0600 (CST) - (envelope-from pgsql-hackers-owner+M16365=candle.pha.pa.us=pgman@postgresql.org) -Received: from mx1.relaypoint.net (ns2.generalbroadband.com [64.32.62.5]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id fB7HQfm90424 - for ; Fri, 7 Dec 2001 12:26:42 -0500 (EST) - (envelope-from joseph.conway@home.com) -Received: from [206.19.64.3] (account jconway@wwc.com HELO home.com) - by mx1.relaypoint.net (CommuniGate Pro SMTP 3.4.8) - with ESMTP id 1722339; Fri, 07 Dec 2001 09:26:46 -0800 -Message-ID: <3C10FBD7.4070602@home.com> -Date: Fri, 07 Dec 2001 09:26:47 -0800 -From: Joe Conway -User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.6+) Gecko/20011126 -X-Accept-Language: en-us -MIME-Version: 1.0 -To: mlw -cc: "Ross J. Reedstrom" , - "pgsql-hackers@postgresql.org" -Subject: Re: [HACKERS] Remote connections? -References: <3C0FB8B4.382C7736@mohawksoft.com> <20011206140616.C10995@rice.edu> <3C0FD339.6F663329@mohawksoft.com> <3C0FEB2C.70000@home.com> <3C104E38.DA19C867@mohawksoft.com> -Content-Type: text/plain; charset=us-ascii; format=flowed -Content-Transfer-Encoding: 7bit -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -mlw wrote: - -> Hey this looks really cool. It looks like something I was thinking about doing. -> I have a couple suggestions that could make it a little better, I hope you will -> not be offended. (If you want my help, I'll chip in!) -> - - -Thanks! Suggestions welcomed. - -> Why not use a binary cursor? That way native types can slip through without the -> overhead of conversion. -> - - -I wasn't sure that would work. Would you create dblink_tok as returning -opaque then? - - -> Right now you get all rows up front, you may be able to increase overall -> performance by fetching only a few rows at a time, rather than get everything -> all at once. (Think on the order of 4 million rows from your remote query!) -> Execute the commit at the end of processing. There are even some asynchronous -> functions you may be able to utilize to reduce the I/O bottleneck. Use the -> synchronous function first, then before you return initiate an asynchronous -> read. Every successive pass through the function, read the newly arrived tuple, -> and initiate the next asynchronous read. (The two machine could be processing -> the query simultaneously, and this could even IMPROVE performance over a single -> system for heavy duty queries.) - - -Interesting . . . but aren't there some issues with the asynch functions? - -> -> Setup a hash table for field names, rather than requiring field numbers. (Keep -> field number for efficiency, of course.) -> -> You could eliminate having to pass the result pointer around by keeping a -> static array in your extension. Use something like Oracle's "contains" notation -> of result number. Where each instantiation of "contains()" and "score()" -> require an id. i.e. 1,2,3,40 etc. Then hash those numbers into an array. I have -> some code that does this for a PostgreSQL extension (it implements contains) on -> my website (pgcontains, under download). It is ugly but works for the most -> part. -> - - -I thought about the static array, but I'm not familiar with Oracle -contains() and score() -- I'm only fluent enough with Oracle to be -dangerous. Guess I'll have to dig out the books . . . - - -> Seriously, your stuff looks great. I think it could be the beginning of a -> fairly usable system for my company. Any help you need/want, just let me know. -> - -I am planning to improve dblink during the next release cycle, so I'll -keep all this in mind (and might take you up on the help offer too!). I -was hoping we'd have functions returning tuples by now, which would -improve this extension dramatically. Unfortunately, it sounds like Alex -won't have time to finish that even for 7.3 :( - -Alex, can we get a look at your latest code? Is it any different the -your last submission to PATCHES? - -Joe - - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate -subscribe-nomail command to majordomo@postgresql.org so that your -message can get through to the mailing list cleanly -