diff --git a/doc/TODO.detail/README b/doc/TODO.detail/README deleted file mode 100644 index 96686217ea..0000000000 --- a/doc/TODO.detail/README +++ /dev/null @@ -1,3 +0,0 @@ -These files are in standard Unix mailbox format, and contain detailed -information related to the TODO list. - diff --git a/doc/TODO.detail/locale b/doc/TODO.detail/locale deleted file mode 100644 index 1bbadc37d6..0000000000 --- a/doc/TODO.detail/locale +++ /dev/null @@ -1,3890 +0,0 @@ -From pgsql-patches-owner+M16987@postgresql.org Thu Aug 4 17:35:52 2005 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j74LZot06577 - for ; Thu, 4 Aug 2005 17:35:50 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id ED433529DB; - Thu, 4 Aug 2005 18:35:44 -0300 (ADT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 70225-08; Thu, 4 Aug 2005 21:35:44 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 0ECB452998; - Thu, 4 Aug 2005 18:35:44 -0300 (ADT) -X-Original-To: pgsql-patches-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id AFAF752840 - for ; Thu, 4 Aug 2005 18:33:25 -0300 (ADT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 70920-01 - for ; - Thu, 4 Aug 2005 21:33:13 +0000 (GMT) -Received: from mx-2.sollentuna.net (mx-2.sollentuna.net [195.84.163.199]) - by svr1.postgresql.org (Postfix) with ESMTP id E1C935280D - for ; Thu, 4 Aug 2005 18:33:11 -0300 (ADT) -Received: from ALGOL.sollentuna.se (janus.sollentuna.se [62.65.68.67]) - by mx-2.sollentuna.net (Postfix) with ESMTP id 7F5A08F289 - for ; Thu, 4 Aug 2005 23:33:13 +0200 (CEST) -content-class: urn:content-classes:message -MIME-Version: 1.0 -Content-Type: multipart/mixed; - boundary="----_=_NextPart_001_01C5993C.1E1CB100" -Subject: [PATCHES] FW: Win32 unicode vs ICU -X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 -Date: Thu, 4 Aug 2005 23:33:12 +0200 -Message-ID: <6BCB9D8A16AC4241919521715F4D8BCE094656@algol.sollentuna.se> -X-MS-Has-Attach: yes -Thread-Topic: Win32 unicode vs ICU -Thread-Index: AcWVyKf6M9GrCqZAQGeSvCZNg9n/jgDc1WwQ -From: "Magnus Hagander" -To: "PostgreSQL-patches" -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-patches -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-patches-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: ORr - -This is a multi-part message in MIME format. - -------_=_NextPart_001_01C5993C.1E1CB100 -Content-Type: text/plain; - charset="us-ascii" -Content-Transfer-Encoding: quoted-printable - -I just realised this mail didn't go through. Probably because it was too -large for -hackers. So: repost to -patches. Sorry about that. If it's a -duplicate, even more sorry, but I couldn't find it in the archives. - -(This may explain that nobody answered me :P) - -//Magnus -=20 - -> -----Original Message----- -> From: Magnus Hagander=20 -> Sent: Sunday, July 31, 2005 2:09 PM -> To: PostgreSQL-development -> Cc: pgsql-hackers-win32@postgresql.org -> Subject: Win32 unicode vs ICU ->=20 -> Hi! ->=20 -> I've been working with Palles ICU patch to make it work on=20 -> win32, and I believe I have it done. While doing it I noticed=20 -> that ICU basically converts to UTF16 and back - I previously=20 -> thought it worked on UTF8 strings. Based on this I also tried=20 -> out an implementation for the win32-unicode problem that does=20 -> *not* require ICU. It uses the win32 native functions to map=20 -> to utf16 and back, and then to process the text there. And I=20 -> got through with much less code than the ICU version, while=20 -> doing the same thing. ->=20 -> I am unsure of how to proceed. As I see it there are three paths: -> 1) Use native win32 functionality only on win32 -> 2) Use ICU functionality only on win32 -> 3) Allow both ICU and native functionality, compile time=20 -> switch --with-icu (same as unix with the ICU patch) ->=20 ->=20 -> The main downsides of ICU vs the native ones are: -> * ICU does not accept win32 locale names. When doing=20 -> setlocale("sv_se"), for example, win32 will return this in=20 -> later calls as "Swedish_Sweden.1252". To get around this in=20 -> the ICU patch, I had to implement a lookup map that converts=20 -> it back to sv_se for ICU. ->=20 -> * ICU is yet another build and runtime dependency, and a=20 -> large one (comes in at 11Mb for the DLL files alone in the=20 -> win32 download) ->=20 ->=20 -> I guess that the main upside of it is that we'd get=20 -> constistent behaviour - in case there are issues with either=20 -> ICU or win32 native they'd otherwise differ. And only one new=20 -> codepath. But we already live with the platform-inconsistency today... ->=20 -> Another upside is that it handles more encodings in ICU - my=20 -> native implementation does *only* UTF8 and relies on existing=20 -> functionality to deal with other encodings. It could of=20 -> course be extended if necessary, but from what I can tell=20 -> UTF8 is the big one. ->=20 ->=20 ->=20 -> I have attached both patches. For the native version, only=20 -> win32_utf8.patch is required. For the ICU version,=20 -> icu_win32.patch is needed and also the files=20 -> localemap.c,localemap.pl, iso639 and iso3166 needs to go in=20 -> src/backend/port/win32. (the localemap needs to be updated to=20 -> do a better-than-linear search, but I wanted to include an example) ->=20 ->=20 -> Thoughts on the options? ->=20 ->=20 -> And anohter question - my native patch touches the same=20 -> functions as the ICU patch. Can somebody who knows the=20 -> internals confirm or deny that these are all the required=20 -> locations, or do we need to modify more? ->=20 -> (I have run simple tests in swedish locale and both behave=20 -> the same and correct, but I'm unsure of exactly how much=20 -> would be affected) ->=20 -> Finally, the win32 patch also changes the normal path to use=20 -> strncoll(). The comment above the function states that we'd=20 -> like to use strncoll but it's not available. Well, on win32=20 -> it is, so it should provide a speedup on win32. It is=20 -> currently not included in the ICU patch, but should probably=20 -> be included whichever path we'd chose. ->=20 ->=20 -> //Magnus ->=20 - -------_=_NextPart_001_01C5993C.1E1CB100 -Content-Type: application/octet-stream; - name="win32_utf8.patch" -Content-Transfer-Encoding: base64 -Content-Description: win32_utf8.patch -Content-Disposition: attachment; - filename="win32_utf8.patch" - -SW5kZXg6IHNyYy9iYWNrZW5kL3V0aWxzL2FkdC9vcmFjbGVfY29tcGF0LmMN -Cj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 -PT09PT09PT09PT09PT09PT09PT09PT0NClJDUyBmaWxlOiAvcHJvamVjdHMv -Y3Zzcm9vdC9wZ3NxbC9zcmMvYmFja2VuZC91dGlscy9hZHQvb3JhY2xlX2Nv -bXBhdC5jLHYNCnJldHJpZXZpbmcgcmV2aXNpb24gMS42MA0KZGlmZiAtYyAt -cjEuNjAgb3JhY2xlX2NvbXBhdC5jDQoqKiogc3JjL2JhY2tlbmQvdXRpbHMv -YWR0L29yYWNsZV9jb21wYXQuYwk3IE1heSAyMDA1IDE1OjE4OjE3IC0wMDAw -CTEuNjANCi0tLSBzcmMvYmFja2VuZC91dGlscy9hZHQvb3JhY2xlX2NvbXBh -dC5jCTMxIEp1bCAyMDA1IDExOjExOjI4IC0wMDAwDQoqKioqKioqKioqKioq -KioNCioqKiAxNDksMTU0ICoqKioNCi0tLSAxNDksMjA5IC0tLS0NCiAgI2Vu -ZGlmICAgLyogVVNFX1dJREVfVVBQRVJfTE9XRVIgKi8NCiAgDQogIA0KKyAj -aWZkZWYgV0lOMzINCisgLyogV2luMzIgVVRGOC9VVEYxNiBjb252ZXJzaW9u -IGhlbHBlcnMgKi8NCisgc3RhdGljIHdjaGFyX3QgKndpbjMyX3V0Zjh0ZXh0 -dG91dGYxNihjb25zdCB0ZXh0ICp0eHQpDQorIHsNCisgCWludAkJCW5ieXRl -cyA9IFZBUlNJWkUodHh0KSAtIFZBUkhEUlNaOwkNCisgCXdjaGFyX3QgICAg -KnJlc3VsdDsNCisgCWludCAgICAgICAgIHI7DQorIA0KKyAJaWYgKG5ieXRl -cyA8IDAgfHwNCisgCQkJbmJ5dGVzID4gKGludCkgKElOVF9NQVggLyBzaXpl -b2Yod2NoYXJfdCkpIC0xKQ0KKyAJCWVyZXBvcnQoRVJST1IsDQorIAkJCQko -ZXJyY29kZShFUlJDT0RFX09VVF9PRl9NRU1PUlkpLA0KKyAJCQkJIGVycm1z -Zygib3V0IG9mIG1lbW9yeSIpKSk7DQorIA0KKyAJLyogT3V0cHV0IHdvcmtz -cGFjZSBjYW5ub3QgaGF2ZSBtb3JlIGNvZGVzIHRoYW4gaW5wdXQgYnl0ZXMg -Ki8NCisgCXJlc3VsdCA9ICh3Y2hhcl90ICopIHBhbGxvYygobmJ5dGVzICsg -MSkgKiBzaXplb2Yod2NoYXJfdCkpOw0KKyANCisgCS8qIERvIHRoZSBjb252 -ZXJzaW9uICovDQorIAlyID0gTXVsdGlCeXRlVG9XaWRlQ2hhcihDUF9VVEY4 -LCAwLCBWQVJEQVRBKHR4dCksICBuYnl0ZXMsDQorIAkJCQlyZXN1bHQsIG5i -eXRlcyAqIHNpemVvZih3Y2hhcl90KSk7DQorIAlpZiAoIXIpDQorIAkJZXJl -cG9ydChFUlJPUiwNCisgCQkJCShlcnJjb2RlKEVSUkNPREVfQ0hBUkFDVEVS -X05PVF9JTl9SRVBFUlRPSVJFKSwNCisgCQkJCSBlcnJtc2coImludmFsaWQg -bXVsdGlieXRlIGNoYXJhY3RlciBmb3IgbG9jYWxlIiksDQorIAkJCQkgZXJy -aGludCgiVGhlIHNlcnZlcidzIExDX0NUWVBFIGxvY2FsZSBpcyBwcm9iYWJs -eSBpbmNvbXBhdGlibGUgd2l0aCB0aGUgZGF0YWJhc2UgZW5jb2RpbmcuIikp -KTsNCisgDQorIAlyZXN1bHRbcl0gPSAwOwkNCisgCXJldHVybiByZXN1bHQ7 -DQorIH0NCisgDQorIHN0YXRpYyB0ZXh0ICp3aW4zMl91dGYxNnRvdXRmOHRl -eHQoY29uc3Qgd2NoYXJfdCAqdHh0KQ0KKyB7DQorIAl0ZXh0CQkqcmVzdWx0 -Ow0KKyAJaW50CQkJIG5ieXRlczsNCisgCWludAkJCSByOw0KKyAJDQorIAlu -Ynl0ZXMgPSBXaWRlQ2hhclRvTXVsdGlCeXRlKENQX1VURjgsIDAsIHR4dCwg -LTEsIE5VTEwsIDAsIE5VTEwsIE5VTEwpOw0KKyAJaWYgKG5ieXRlcyA9PSAw -KQ0KKyAJCWVyZXBvcnQoRVJST1IsDQorIAkJCQkoZXJyY29kZShFUlJDT0RF -X0NIQVJBQ1RFUl9OT1RfSU5fUkVQRVJUT0lSRSksDQorIAkJCQkgZXJybXNn -KCJpbnZhbGlkIHV0ZjE2IGNoYXJhY3RlciBmb3IgbG9jYWxlOiAlbHUiLEdl -dExhc3RFcnJvcigpKSkpOw0KKyANCisgCXJlc3VsdCA9IHBhbGxvYyhuYnl0 -ZXMrVkFSSERSU1opOw0KKyANCisgCXIgPSBXaWRlQ2hhclRvTXVsdGlCeXRl -KENQX1VURjgsIDAsIHR4dCwgLTEsIFZBUkRBVEEocmVzdWx0KSwgbmJ5dGVz -LCBOVUxMLCBOVUxMKTsNCisgCWlmIChyID09IDApDQorIAkJZXJlcG9ydChF -UlJPUiwNCisgCQkJCShlcnJjb2RlKEVSUkNPREVfQ0hBUkFDVEVSX05PVF9J -Tl9SRVBFUlRPSVJFKSwNCisgCQkJCSBlcnJtc2coImludmFsaWQgdXRmMTYg -Y2hhcmFjdGVyIGZvciBsb2NhbGU6ICVsdSIsR2V0TGFzdEVycm9yKCkpKSk7 -DQorIA0KKyAJVkFSQVRUX1NJWkVQKHJlc3VsdCkgPSBuYnl0ZXMgKyBWQVJI -RFJTWiAtIDE7IC8qIC0xID0gaWdub3JlIG51bGwgKi8NCisgCXJldHVybiBy -ZXN1bHQ7DQorIH0NCisgI2VuZGlmIC8qIFdJTjMyICovDQorIA0KICAvKioq -KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq -KioqKioqKioqKioqKioqKioqKioNCiAgICoNCiAgICogbG93ZXINCioqKioq -KioqKioqKioqKg0KKioqIDE3OSwxODQgKioqKg0KLS0tIDIzNCwyNTEgLS0t -LQ0KICAJCXdjaGFyX3QgICAgKndvcmtzcGFjZTsNCiAgCQlpbnQJCQlpOw0K -ICANCisgI2lmZGVmIFdJTjMyDQorIAkJLyogV2luMzIgZG9lcyBub3QgaGF2 -ZSBVVEYtOCwgc28gd2UgbmVlZCB0byBtYXAgdG8gVVRGLTE2ICovDQorIAkJ -aWYgKEdldERhdGFiYXNlRW5jb2RpbmcoKSA9PSBQR19VVEY4KQ0KKyAJCXsN -CisgCQkJd29ya3NwYWNlID0gd2luMzJfdXRmOHRleHR0b3V0ZjE2KHN0cmlu -Zyk7DQorIAkJCV93Y3Nsd3Iod29ya3NwYWNlKTsNCisgCQkJcmVzdWx0ID0g -d2luMzJfdXRmMTZ0b3V0Zjh0ZXh0KHdvcmtzcGFjZSk7DQorIAkJCXBmcmVl -KHdvcmtzcGFjZSk7DQorIAkJCVBHX1JFVFVSTl9URVhUX1AocmVzdWx0KTsN -CisgCQl9DQorICNlbmRpZg0KKyAJCQ0KICAJCXdvcmtzcGFjZSA9IHRleHR0 -b3djcyhzdHJpbmcpOw0KICANCiAgCQlmb3IgKGkgPSAwOyB3b3Jrc3BhY2Vb -aV0gIT0gMDsgaSsrKQ0KKioqKioqKioqKioqKioqDQoqKiogMjQ1LDI1MCAq -KioqDQotLS0gMzEyLDMyOSAtLS0tDQogIAkJd2NoYXJfdCAgICAqd29ya3Nw -YWNlOw0KICAJCWludAkJCWk7DQogIA0KKyAjaWZkZWYgV0lOMzINCisgCQkv -KiBXaW4zMiBkb2VzIG5vdCBoYXZlIFVURi04LCBzbyB3ZSBuZWVkIHRvIG1h -cCB0byBVVEYtMTYgKi8NCisgCQlpZiAoR2V0RGF0YWJhc2VFbmNvZGluZygp -ID09IFBHX1VURjgpDQorIAkJew0KKyAJCQl3b3Jrc3BhY2UgPSB3aW4zMl91 -dGY4dGV4dHRvdXRmMTYoc3RyaW5nKTsNCisgCQkJX3djc3Vwcih3b3Jrc3Bh -Y2UpOw0KKyAJCQlyZXN1bHQgPSB3aW4zMl91dGYxNnRvdXRmOHRleHQod29y -a3NwYWNlKTsNCisgCQkJcGZyZWUod29ya3NwYWNlKTsNCisgCQkJUEdfUkVU -VVJOX1RFWFRfUChyZXN1bHQpOw0KKyAJCX0NCisgI2VuZGlmDQorIA0KICAJ -CXdvcmtzcGFjZSA9IHRleHR0b3djcyhzdHJpbmcpOw0KICANCiAgCQlmb3Ig -KGkgPSAwOyB3b3Jrc3BhY2VbaV0gIT0gMDsgaSsrKQ0KKioqKioqKioqKioq -KioqDQoqKiogMzE1LDMyMCAqKioqDQotLS0gMzk0LDQwNSAtLS0tDQogIAkJ -aW50CQkJd2FzYWxudW0gPSAwOw0KICAJCWludAkJCWk7DQogIA0KKyAjaWZk -ZWYgV0lOMzINCisgCQkvKiBXaW4zMiBkb2VzIG5vdCBoYXZlIFVURi04LCBz -byB3ZSBuZWVkIHRvIG1hcCB0byBVVEYtMTYgKi8NCisgCQlpZiAoR2V0RGF0 -YWJhc2VFbmNvZGluZygpID09IFBHX1VURjgpDQorIAkJCXdvcmtzcGFjZSA9 -IHdpbjMyX3V0Zjh0ZXh0dG91dGYxNihzdHJpbmcpOw0KKyAJCWVsc2UNCisg -I2VuZGlmDQogIAkJd29ya3NwYWNlID0gdGV4dHRvd2NzKHN0cmluZyk7DQog -IA0KICAJCWZvciAoaSA9IDA7IHdvcmtzcGFjZVtpXSAhPSAwOyBpKyspDQoq -KioqKioqKioqKioqKioNCioqKiAzMjYsMzMxICoqKioNCi0tLSA0MTEsNDIy -IC0tLS0NCiAgCQkJd2FzYWxudW0gPSBpc3dhbG51bSh3b3Jrc3BhY2VbaV0p -Ow0KICAJCX0NCiAgDQorICNpZmRlZiBXSU4zMg0KKyAJCS8qIE1hcCBiYWNr -IHRvIFVURi04ICovDQorIAkJaWYgKEdldERhdGFiYXNlRW5jb2RpbmcoKSA9 -PSBQR19VVEY4KQ0KKyAJCQlyZXN1bHQgPSB3aW4zMl91dGYxNnRvdXRmOHRl -eHQod29ya3NwYWNlKTsNCisgCQllbHNlDQorICNlbmRpZg0KICAJCXJlc3Vs -dCA9IHdjc3RvdGV4dCh3b3Jrc3BhY2UsIGkpOw0KICANCiAgCQlwZnJlZSh3 -b3Jrc3BhY2UpOw0KSW5kZXg6IHNyYy9iYWNrZW5kL3V0aWxzL2FkdC92YXJs -ZW5hLmMNCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 -PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0NClJDUyBmaWxlOiAvcHJv -amVjdHMvY3Zzcm9vdC9wZ3NxbC9zcmMvYmFja2VuZC91dGlscy9hZHQvdmFy -bGVuYS5jLHYNCnJldHJpZXZpbmcgcmV2aXNpb24gMS4xMzANCmRpZmYgLWMg -LXIxLjEzMCB2YXJsZW5hLmMNCioqKiBzcmMvYmFja2VuZC91dGlscy9hZHQv -dmFybGVuYS5jCTI5IEp1bCAyMDA1IDAzOjE3OjU1IC0wMDAwCTEuMTMwDQot -LS0gc3JjL2JhY2tlbmQvdXRpbHMvYWR0L3ZhcmxlbmEuYwkzMSBKdWwgMjAw -NSAxMToxNTo0OSAtMDAwMA0KKioqKioqKioqKioqKioqDQoqKiogODQ4LDg1 -MyAqKioqDQotLS0gODQ4LDkwMSAtLS0tDQogIAkJY2hhcgkJYTJidWZbU1RB -Q0tCVUZMRU5dOw0KICAJCWNoYXIJICAgKmExcCwNCiAgCQkJCSAgICphMnA7 -DQorICNpZmRlZiBXSU4zMg0KKyAJCS8qIFdpbjMyIGRvZXMgbm90IGhhdmUg -VVRGLTgsIHNvIHdlIG5lZWQgdG8gbWFwIHRvIFVURi0xNiAqLw0KKyAJCWlm -IChHZXREYXRhYmFzZUVuY29kaW5nKCkgPT0gUEdfVVRGOCkNCisgCQl7DQor -IAkJCWludCBhMWxlbiA9IFNUQUNLQlVGTEVOOw0KKyAJCQlpbnQgYTJsZW4g -PSBTVEFDS0JVRkxFTjsNCisgDQorIAkJCWlmIChsZW4xID49IFNUQUNLQlVG -TEVOLzIpDQorIAkJCXsNCisgCQkJCWExbGVuID0gbGVuMSAqIDIgKyAyOw0K -KyAJCQkJYTFwID0gcGFsbG9jKGExbGVuKTsNCisgCQkJfQ0KKyAJCQllbHNl -DQorIAkJCQlhMXAgPSBhMWJ1ZjsNCisgCQkJDQorIAkJCWlmIChsZW4yID49 -IFNUQUNLQlVGTEVOLzIpDQorIAkJCXsNCisgCQkJCWEybGVuID0gbGVuMiAq -IDIgKyAyOw0KKyAJCQkJYTJwID0gcGFsbG9jKGEybGVuKTsNCisgCQkJfQ0K -KyAJCQllbHNlDQorIAkJCQlhMnAgPSBhMmJ1ZjsNCisgDQorIAkJCWlmICgh -TXVsdGlCeXRlVG9XaWRlQ2hhcihDUF9VVEY4LCAwLCBhcmcxLCAtMSwgKExQ -V1NUUilhMXAsIGExbGVuLzIpKQ0KKyAJCQkJZXJlcG9ydChFUlJPUiwNCisg -CQkJCQkJKGVycm1zZygiZmFpbGVkIHRvIGNvbnZlcnQgc3RyaW5nIHRvIFVU -RjE2OiAlbHUiLA0KKyAJCQkJCQkJR2V0TGFzdEVycm9yKCkpKSk7DQorIAkJ -CWlmICghTXVsdGlCeXRlVG9XaWRlQ2hhcihDUF9VVEY4LCAwLCBhcmcyLCAt -MSwgKExQV1NUUilhMnAsIGEybGVuLzIpKQ0KKyAJCQkJZXJlcG9ydChFUlJP -UiwNCisgCQkJCQkJKGVycm1zZygiZmFpbGVkIHRvIGNvbnZlcnQgc3RyaW5n -IHRvIFVURjE2OiAlbHUiLA0KKyAJCQkJCQkJR2V0TGFzdEVycm9yKCkpKSk7 -DQorIAkJCQ0KKyAJCQllcnJubz0wOw0KKyAJCQlyZXN1bHQgPSB3Y3Njb2xs -KChMUFdTVFIpYTFwLCAoTFBXU1RSKWEycCk7DQorIAkJCWlmIChyZXN1bHQg -PT0gMjE0NzQ4MzY0NykgLyogX05MU0NNUEVSUk9SOyBtaXNzaW5nIGZyb20g -bWluZ3cgaGVhZGVycyAqLw0KKyAJCQkJZXJlcG9ydChFUlJPUiwNCisgCQkJ -CQkJKGVycm1zZygiZmFpbGVkIHRvIGNvbXBhcmUgdW5pY29kZSBzdHJpbmdz -OiAlaSIsIGVycm5vKSkpOw0KKyAJCQlpZiAoYTFsZW4gIT0gU1RBQ0tCVUZM -RU4pDQorIAkJCQlwZnJlZShhMXApOw0KKyAJCQlpZiAoYTJsZW4gIT0gU1RB -Q0tCVUZMRU4pDQorIAkJCQlwZnJlZShhMnApOw0KKyAJCQkNCisgCQkJcmV0 -dXJuIHJlc3VsdDsNCisgCQl9DQorIA0KKyAJCS8qIFdpbjMyIGhhcyBzdHJu -Y29sbCgpLCBzbyB1c2UgaXQgdGhlcmUgdG8gYXZvaWQgY29weWluZyAqLw0K -KyAJCXJldHVybiBfc3RybmNvbGwoYXJnMSwgYXJnMiwgKGxlbjE+bGVuMik/ -bGVuMjpsZW4xICk7DQorICNlbmRpZg0KICANCiAgCQlpZiAobGVuMSA+PSBT -VEFDS0JVRkxFTikNCiAgCQkJYTFwID0gKGNoYXIgKikgcGFsbG9jKGxlbjEg -KyAxKTsNCg== - -------_=_NextPart_001_01C5993C.1E1CB100 -Content-Type: application/octet-stream; - name="icu_win32.patch" -Content-Transfer-Encoding: base64 -Content-Description: icu_win32.patch -Content-Disposition: attachment; - filename="icu_win32.patch" - -SW5kZXg6IGNvbmZpZ3VyZS5pbg0KPT09PT09PT09PT09PT09PT09PT09PT09 -PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQ0K -UkNTIGZpbGU6IC9wcm9qZWN0cy9jdnNyb290L3Bnc3FsL2NvbmZpZ3VyZS5p -bix2DQpyZXRyaWV2aW5nIHJldmlzaW9uIDEuNDE3DQpkaWZmIC1jIC1yMS40 -MTcgY29uZmlndXJlLmluDQoqKiogY29uZmlndXJlLmluCTYgSnVsIDIwMDUg -MjE6MDQ6MTMgLTAwMDAJMS40MTcNCi0tLSBjb25maWd1cmUuaW4JMjggSnVs -IDIwMDUgMTc6MDE6NDcgLTAwMDANCioqKioqKioqKioqKioqKg0KKioqIDQ2 -Nyw0NzIgKioqKg0KLS0tIDQ2Nyw0ODAgLS0tLQ0KICBBQ19NU0dfUkVTVUxU -KFskd2l0aF9vcGVuc3NsXSkNCiAgQUNfU1VCU1Qod2l0aF9vcGVuc3NsKQ0K -ICANCisgIw0KKyAjIElDVQ0KKyAjDQorIEFDX01TR19DSEVDS0lORyhbd2hl -dGhlciB0byBidWlsZCB3aXRoIElDVSBzdXBwb3J0XSkNCisgUEdBQ19BUkdf -Qk9PTCh3aXRoLCBpY3UsIG5vLCBbICAtLXdpdGgtaWN1ICAgICAgICAgICAg -ICBidWlsZCB3aXRoIElDVSBzdXBwb3J0XSwNCisgICAgICAgICAgICAgICBb -QUNfREVGSU5FKFtVU0VfSUNVXSwgMSwgW0RlZmluZSB0byBidWlsZCB3aXRo -IElDVSBzdXBwb3J0LiAoLS13aXRoLWljdSldKV0pDQorIEFDX01TR19SRVNV -TFQoWyR3aXRoX2ljdV0pDQorIEFDX1NVQlNUKHdpdGhfaWN1KQ0KICANCiAg -Iw0KICAjIFJlYWRsaW5lDQoqKioqKioqKioqKioqKioNCioqKiA2NzQsNjc5 -ICoqKioNCi0tLSA2ODIsNjk4IC0tLS0NCiAgICBmaQ0KICBmaQ0KICANCisg -aWYgdGVzdCAiJHdpdGhfaWN1IiA9IHllcyA7IHRoZW4NCisgICBpZiB0ZXN0 -ICIkUE9SVE5BTUUiICE9ICJ3aW4zMiI7IHRoZW4NCisgICAgIEFDX0NIRUNL -X0xJQihpY3VpMThuLCB1Y29sX29wZW5fM18yLCBbXSwgW0FDX01TR19FUlJP -UihbbGlicmFyeSAnaWN1aTE4bicgaXMgcmVxdWlyZWQgZm9yIElDVV0pXSkN -CisgICAgIEFDX0NIRUNLX0xJQihpY3V1YywgIHVfdG9sb3dlcl8zXzIsIFtd -LCBbQUNfTVNHX0VSUk9SKFtsaWJyYXJ5ICdpY3V1YycgaXMgcmVxdWlyZWQg -Zm9yIElDVV0pXSkNCisgICAgIEFDX0NIRUNLX0xJQihpY3VkYXRhLCBpY3Vk -dDMyX2RhdCwgW10sIFtBQ19NU0dfRVJST1IoW2xpYnJhcnkgJ2ljdWRhdGEn -IGlzIHJlcXVpcmVkIGZvciBJQ1VdKV0pDQorICAgZWxzZQ0KKyAgICAgQUNf -Q0hFQ0tfTElCKGljdWluLCB1Y29sX29wZW5fM18yLCBbXSwgW0FDX01TR19F -UlJPUihbbGlicmFyeSAnaWN1aW4nIGlzIHJlcXVpcmVkIGZvciBJQ1VdKV0p -DQorICAgICBBQ19DSEVDS19MSUIoaWN1dWMsICB1X3RvbG93ZXJfM18yLCBb -XSwgW0FDX01TR19FUlJPUihbbGlicmFyeSAnaWN1dWMnIGlzIHJlcXVpcmVk -IGZvciBJQ1VdKV0pDQorICAgZmkNCisgZmkNCisgICANCiAgaWYgdGVzdCAi -JHdpdGhfcGFtIiA9IHllcyA7IHRoZW4NCiAgICBBQ19DSEVDS19MSUIocGFt -LCAgICBwYW1fc3RhcnQsIFtdLCBbQUNfTVNHX0VSUk9SKFtsaWJyYXJ5ICdw -YW0nIGlzIHJlcXVpcmVkIGZvciBQQU1dKV0pDQogIGZpDQoqKioqKioqKioq -KioqKioNCioqKiA3NDgsNzUzICoqKioNCi0tLSA3NjcsNzc2IC0tLS0NCiAg -ICBBQ19DSEVDS19IRUFERVIob3BlbnNzbC9lcnIuaCwgW10sIFtBQ19NU0df -RVJST1IoW2hlYWRlciBmaWxlIDxvcGVuc3NsL2Vyci5oPiBpcyByZXF1aXJl -ZCBmb3IgT3BlblNTTF0pXSkNCiAgZmkNCiAgDQorIGlmIHRlc3QgIiR3aXRo -X2ljdSIgPSB5ZXMgOyB0aGVuDQorICAgQUNfQ0hFQ0tfSEVBREVSKHVuaWNv -ZGUvdXR5cGVzLmgsIFtdLCBbQUNfTVNHX0VSUk9SKFtoZWFkZXIgZmlsZSA8 -dW5pY29kZS91dHlwZXMuaD4gaXMgcmVxdWlyZWQgZm9yIElDVV0pXSkNCisg -ZmkNCisgIA0KICBpZiB0ZXN0ICIkd2l0aF9wYW0iID0geWVzIDsgdGhlbg0K -ICAgIEFDX0NIRUNLX0hFQURFUlMoc2VjdXJpdHkvcGFtX2FwcGwuaCwgW10s -DQogICAgICAgICAgICAgICAgICAgICBbQUNfQ0hFQ0tfSEVBREVSUyhwYW0v -cGFtX2FwcGwuaCwgW10sDQpJbmRleDogc3JjL2JhY2tlbmQvcG9ydC93aW4z -Mi9NYWtlZmlsZQ0KPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 -PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQ0KUkNTIGZpbGU6 -IC9wcm9qZWN0cy9jdnNyb290L3Bnc3FsL3NyYy9iYWNrZW5kL3BvcnQvd2lu -MzIvTWFrZWZpbGUsdg0KcmV0cmlldmluZyByZXZpc2lvbiAxLjYNCmRpZmYg -LWMgLXIxLjYgTWFrZWZpbGUNCioqKiBzcmMvYmFja2VuZC9wb3J0L3dpbjMy -L01ha2VmaWxlCTI5IEF1ZyAyMDA0IDAwOjM4OjAzIC0wMDAwCTEuNg0KLS0t -IHNyYy9iYWNrZW5kL3BvcnQvd2luMzIvTWFrZWZpbGUJMzAgSnVsIDIwMDUg -MTc6NTI6NDcgLTAwMDANCioqKioqKioqKioqKioqKg0KKioqIDEyLDI5ICoq -KioNCiAgdG9wX2J1aWxkZGlyID0gLi4vLi4vLi4vLi4NCiAgaW5jbHVkZSAk -KHRvcF9idWlsZGRpcikvc3JjL01ha2VmaWxlLmdsb2JhbA0KICANCiEgT0JK -UyA9IHNlbWEubyBzaG1lbS5vIHRpbWVyLm8gc29ja2V0Lm8gc2lnbmFsLm8g -c2VjdXJpdHkubyBlcnJvci5vDQogIA0KICBhbGw6IFNVQlNZUy5vDQogIA0K -ICBTVUJTWVMubzogJChPQkpTKQ0KICAJJChMRCkgJChMRFJFTCkgJChMRE9V -VCkgU1VCU1lTLm8gJChPQkpTKQ0KICANCiAgZGVwZW5kIGRlcDoNCiAgCSQo -Q0MpIC1NTSAkKENGTEFHUykgKi5jID5kZXBlbmQNCiAgDQogIGNsZWFuOiAN -CiEgCXJtIC1mIFNVQlNZUy5vICQoT0JKUykNCiAgDQogIGlmZXEgKGRlcGVu -ZCwkKHdpbGRjYXJkIGRlcGVuZCkpDQogIGluY2x1ZGUgZGVwZW5kDQotLS0g -MTIsMzQgLS0tLQ0KICB0b3BfYnVpbGRkaXIgPSAuLi8uLi8uLi8uLg0KICBp -bmNsdWRlICQodG9wX2J1aWxkZGlyKS9zcmMvTWFrZWZpbGUuZ2xvYmFsDQog -IA0KISBPQkpTID0gc2VtYS5vIHNobWVtLm8gdGltZXIubyBzb2NrZXQubyBz -aWduYWwubyBzZWN1cml0eS5vIGVycm9yLm8gbG9jYWxlbWFwLm8NCiAgDQog -IGFsbDogU1VCU1lTLm8NCiAgDQogIFNVQlNZUy5vOiAkKE9CSlMpDQogIAkk -KExEKSAkKExEUkVMKSAkKExET1VUKSBTVUJTWVMubyAkKE9CSlMpDQogIA0K -KyBsb2NhbGVtYXAubzogbG9jYWxlbWFwLmMgbG9jYWxlbWFwLmgNCisgDQor -IGxvY2FsZW1hcC5oOiBpc282MzkgaXNvMzE2NiBsb2NhbGVtYXAucGwNCisg -CSQoUEVSTCkgbG9jYWxlbWFwLnBsID4gbG9jYWxlbWFwLmgNCisgDQogIGRl -cGVuZCBkZXA6DQogIAkkKENDKSAtTU0gJChDRkxBR1MpICouYyA+ZGVwZW5k -DQogIA0KICBjbGVhbjogDQohIAlybSAtZiBTVUJTWVMubyAkKE9CSlMpIGxv -Y2FsZW1hcC5oDQogIA0KICBpZmVxIChkZXBlbmQsJCh3aWxkY2FyZCBkZXBl -bmQpKQ0KICBpbmNsdWRlIGRlcGVuZA0KSW5kZXg6IHNyYy9iYWNrZW5kL3V0 -aWxzL2FkdC9vcmFjbGVfY29tcGF0LmMNCj09PT09PT09PT09PT09PT09PT09 -PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 -PT0NClJDUyBmaWxlOiAvcHJvamVjdHMvY3Zzcm9vdC9wZ3NxbC9zcmMvYmFj -a2VuZC91dGlscy9hZHQvb3JhY2xlX2NvbXBhdC5jLHYNCnJldHJpZXZpbmcg -cmV2aXNpb24gMS42MA0KZGlmZiAtYyAtcjEuNjAgb3JhY2xlX2NvbXBhdC5j -DQoqKiogc3JjL2JhY2tlbmQvdXRpbHMvYWR0L29yYWNsZV9jb21wYXQuYwk3 -IE1heSAyMDA1IDE1OjE4OjE3IC0wMDAwCTEuNjANCi0tLSBzcmMvYmFja2Vu -ZC91dGlscy9hZHQvb3JhY2xlX2NvbXBhdC5jCTMwIEp1bCAyMDA1IDE0OjU5 -OjI5IC0wMDAwDQoqKioqKioqKioqKioqKioNCioqKiAzMiwzNyAqKioqDQot -LS0gMzIsNDMgLS0tLQ0KICAjaW5jbHVkZSAidXRpbHMvcGdfbG9jYWxlLmgi -DQogICNpbmNsdWRlICJtYi9wZ193Y2hhci5oIg0KICANCisgI2lmZGVmIFVT -RV9JQ1UNCisgI2luY2x1ZGUgPHVuaWNvZGUvdXR5cGVzLmg+ICAgLyogQmFz -aWMgSUNVIGRhdGEgdHlwZXMgKi8NCisgI2luY2x1ZGUgPHVuaWNvZGUvdWNu -di5oPiAgICAgLyogQyAgIENvbnZlcnRlciBBUEkgICAgKi8NCisgI2luY2x1 -ZGUgPHVuaWNvZGUvdXN0cmluZy5oPg0KKyAjZW5kaWYgLyogVVNFX0lDVSAq -Lw0KKyANCiAgDQogIC8qDQogICAqIElmIHRoZSBzeXN0ZW0gcHJvdmlkZXMg -dGhlIG5lZWRlZCBmdW5jdGlvbnMgZm9yIHdpZGUtY2hhcmFjdGVyIG1hbmlw -dWxhdGlvbg0KKioqKioqKioqKioqKioqDQoqKiogNTMsNTggKioqKg0KLS0t -IDU5LDEwNiAtLS0tDQogIAkgICBib29sIGRvbHRyaW0sIGJvb2wgZG9ydHJp -bSk7DQogIA0KICANCisgI2lmZGVmIFVTRV9JQ1UNCisgc3RhdGljIFVDb252 -ZXJ0ZXIgKmNvbnYgPSBOVUxMOw0KKyANCisgc3RhdGljIHRleHQgKg0KKyBV -Q2hhcnRvdGV4dChjb25zdCBVQ2hhciAqc3RyLCBpbnQgbmNvZGVzKQ0KKyB7 -DQorIAl0ZXh0CSAgICpyZXN1bHQ7DQorIAlzaXplX3QJCW5ieXRlcywgcmVz -dWx0c2l6ZTsNCisgDQorIAlVRXJyb3JDb2RlIHN0YXR1cyA9IFVfWkVST19F -UlJPUjsNCisgDQorIAkvKiBPdmVyZmxvdyBwYXJhbm9pYSAqLw0KKyAJaWYg -KG5jb2RlcyA8IDAgfHwNCisgCQluY29kZXMgPiAoaW50KSAoKElOVF9NQVgg -LSBWQVJIRFJTWikgLyBzaXplb2YoVUNoYXIpKSAtIDEpDQorIAkJZXJlcG9y -dChFUlJPUiwNCisgCQkJCShlcnJjb2RlKEVSUkNPREVfT1VUX09GX01FTU9S -WSksDQorIAkJCQkgZXJybXNnKCJvdXQgb2YgbWVtb3J5IikpKTsNCisgDQor -IAkvKiBNYWtlIHdvcmtzcGFjZSBjZXJ0YWlubHkgbGFyZ2UgZW5vdWdoIGZv -ciByZXN1bHQgKi8NCisgCXJlc3VsdHNpemUgPSBVQ05WX0dFVF9NQVhfQllU -RVNfRk9SX1NUUklORyhuY29kZXMsIHVjbnZfZ2V0TWF4Q2hhclNpemUoY29u -dikpOw0KKyAJcmVzdWx0ID0gKHRleHQgKikgcGFsbG9jKHJlc3VsdHNpemUg -KyBWQVJIRFJTWik7DQorIA0KKyAJbmJ5dGVzID0gdWNudl9mcm9tVUNoYXJz -KGNvbnYsIFZBUkRBVEEocmVzdWx0KSwgcmVzdWx0c2l6ZSwNCisgCQkJCQkJ -CSBzdHIsIG5jb2RlcywgJnN0YXR1cyk7DQorIA0KKyAJaWYgKFVfRkFJTFVS -RShzdGF0dXMpKQ0KKyAJew0KKyAJCS8qIEludmFsaWQgbXVsdGlieXRlIGNo -YXJhY3RlciBlbmNvdW50ZXJlZCAuLi4gc2hvdWxkbid0IGhhcHBlbiAqLw0K -KyAJCWVyZXBvcnQoRVJST1IsDQorIAkJCQkoZXJyY29kZShFUlJDT0RFX0NI -QVJBQ1RFUl9OT1RfSU5fUkVQRVJUT0lSRSksDQorIAkJCQkgZXJybXNnKCJp -bnZhbGlkIG11bHRpYnl0ZSBjaGFyYWN0ZXIgZm9yIGxvY2FsZSIpKSk7DQor -IAl9DQorIA0KKyAJQXNzZXJ0KG5ieXRlcyA8PSAoc2l6ZV90KSAobmNvZGVz -ICogc2l6ZW9mKFVDaGFyKSkpOw0KKyANCisgCVZBUkFUVF9TSVpFUChyZXN1 -bHQpID0gbmJ5dGVzICsgVkFSSERSU1o7DQorIA0KKyAJcmV0dXJuIHJlc3Vs -dDsNCisgfQ0KKyANCisgDQorICNlbHNlDQogICNpZmRlZiBVU0VfV0lERV9V -UFBFUl9MT1dFUg0KICANCiAgLyoNCioqKioqKioqKioqKioqKg0KKioqIDE0 -NywxNTIgKioqKg0KLS0tIDE5NSwyMDEgLS0tLQ0KICAJcmV0dXJuIHJlc3Vs -dDsNCiAgfQ0KICAjZW5kaWYgICAvKiBVU0VfV0lERV9VUFBFUl9MT1dFUiAq -Lw0KKyAjZW5kaWYgICAvKiBVU0VfSUNVICovDQogIA0KICANCiAgLyoqKioq -KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq -KioqKioqKioqKioqKioqKioqDQoqKioqKioqKioqKioqKioNCioqKiAxNjYs -MTcxICoqKioNCi0tLSAyMTUsMjg2IC0tLS0NCiAgRGF0dW0NCiAgbG93ZXIo -UEdfRlVOQ1RJT05fQVJHUykNCiAgew0KKyAjaWZkZWYgVVNFX0lDVQ0KKyAj -ZGVmaW5lIFNUQUNLQlVGTEVOCQkxMDI0IC8gc2l6ZW9mKFVDaGFyKQ0KKyAJ -LyogdXNlIElDVSBvbmx5IHdoZW4gbWF4IGVuY29kaW5nIGxlbmd0aCA+IG9u -ZSAqLw0KKyAJaWYgKHBnX2RhdGFiYXNlX2VuY29kaW5nX21heF9sZW5ndGgo -KSA+IDEpDQorIAl7DQorIAkJdGV4dAkgICAqc3RyaW5nID0gUEdfR0VUQVJH -X1RFWFRfUCgwKTsNCisgCQl0ZXh0CSAgICpyZXN1bHQ7DQorIAkJVUNoYXIg -ICAgICAgc291cmNlYnVmW1NUQUNLQlVGTEVOXSwgZGVzdGJ1ZltTVEFDS0JV -RkxFTl07DQorIAkJVUNoYXIgICAgICAqc291cmNlLCAqZGVzdDsNCisgCQlp -bnQJCQlidWZsZW4sDQorIAkJCQkJYXJnbGVuID0gVkFSU0laRShzdHJpbmcp -IC0gVkFSSERSU1o7DQorIAkJVUVycm9yQ29kZSAgc3RhdHVzID0gVV9aRVJP -X0VSUk9SOw0KKyANCisgCQlpZiAoY29udiA9PSBOVUxMKQ0KKyAJCXsNCisg -CQkJY29udiA9IHVjbnZfb3BlbihOVUxMLCAmc3RhdHVzKTsNCisgCQkJaWYg -KFVfRkFJTFVSRShzdGF0dXMpKQ0KKyAJCQl7DQorIAkJCQllcmVwb3J0KEVS -Uk9SLA0KKyAJCQkJCQkoZXJyY29kZShzdGF0dXMpLA0KKyAJCQkJCQkgZXJy -bXNnKCJJQ1UgZXJyb3I6IG9yYWNsZV9jb21wYXQuYywgY291bGQgbm90IGdl -dCBjb252ZXJ0ZXIgZm9yIFwiJXNcIiIsIHVjbnZfZ2V0RGVmYXVsdE5hbWUo -KSkpKTsNCisgCQkJfQ0KKyAJCX0NCisgDQorIAkJaWYgKGFyZ2xlbiA+PSBT -VEFDS0JVRkxFTiAvIHNpemVvZihVQ2hhcikpDQorIAkJew0KKyAJCQlidWZs -ZW4gPSBhcmdsZW4gKiBzaXplb2YoVUNoYXIpICsgMTsNCisgIAkJCXNvdXJj -ZSA9IHBhbGxvYyhidWZsZW4pOw0KKyAJCQlkZXN0ID0gcGFsbG9jKGJ1Zmxl -bik7DQorIAkJfQ0KKyAJCWVsc2UNCisgCQl7DQorIAkJCWJ1ZmxlbiA9IFNU -QUNLQlVGTEVOOw0KKyAJCQlzb3VyY2UgPSBzb3VyY2VidWY7DQorIAkJCWRl -c3QgPSBkZXN0YnVmOw0KKyAJCX0NCisgCQkvLyBjb252ZXJ0IHRvIFVURi0x -Ng0KKyAJCXVjbnZfdG9VQ2hhcnMoY29udiwgc291cmNlLCBidWZsZW4sIFZB -UkRBVEEoc3RyaW5nKSwgYXJnbGVuLCAmc3RhdHVzKTsNCisgCQlpZiAoVV9G -QUlMVVJFKHN0YXR1cykpDQorIAkJew0KKyAJCQllcmVwb3J0KEVSUk9SLA0K -KyAJCQkJCQkoZXJyY29kZShzdGF0dXMpLA0KKyAJCQkJCQkgZXJybXNnKCJJ -Q1UgZXJyb3I6IENvdWxkIG5vdCBjb252ZXJ0IHN0cmluZyIpKSk7DQorIAkJ -fQ0KKyAJCQ0KKyAJCS8vIHJ1biBkZXNpcmVkIGZ1bmN0aW9uDQorIAkJYnVm -bGVuID0gdV9zdHJUb0xvd2VyKGRlc3QsIGJ1Zmxlbiwgc291cmNlLCAtMSwg -TlVMTCwgJnN0YXR1cyk7DQorIAkJaWYgKFVfRkFJTFVSRShzdGF0dXMpKQ0K -KyAJCXsNCisgCQkJZXJlcG9ydChFUlJPUiwNCisgCQkJCQkJKGVycmNvZGUo -c3RhdHVzKSwNCisgCQkJCQkJIGVycm1zZygiSUNVIGVycm9yOiBDb3VsZCBu -b3QgbW9kaWZ5IGNhc2UiKSkpOw0KKyAJCX0NCisgDQorIAkJLy8gYW5kIGNv -bnZlcnQgbW9kaWZpZWQgdXRmLTE2IHN0cmluZyBiYWNrIHRvIHRleHQNCisg -CQlyZXN1bHQgPSBVQ2hhcnRvdGV4dChkZXN0LCBidWZsZW4pOw0KKyANCisg -CQlpZiAoYXJnbGVuID49IFNUQUNLQlVGTEVOIC8gc2l6ZW9mKFVDaGFyKSkN -CisgCQl7DQorIAkJCXBmcmVlKHNvdXJjZSk7DQorIAkJCXBmcmVlKGRlc3Qp -Ow0KKyAJCX0NCisgCQlQR19SRVRVUk5fVEVYVF9QKHJlc3VsdCk7DQorIAl9 -DQorIAllbHNlDQorICNlbHNlDQogICNpZmRlZiBVU0VfV0lERV9VUFBFUl9M -T1dFUg0KICAJLyoNCiAgCSAqCVVzZSB3aWRlIGNoYXIgY29kZSBvbmx5IHdo -ZW4gbWF4IGVuY29kaW5nIGxlbmd0aCA+IDEgYW5kIGN0eXBlICE9IEMuDQoq -KioqKioqKioqKioqKioNCioqKiAxOTIsMTk3ICoqKioNCi0tLSAzMDcsMzEz -IC0tLS0NCiAgCX0NCiAgCWVsc2UNCiAgI2VuZGlmICAgLyogVVNFX1dJREVf -VVBQRVJfTE9XRVIgKi8NCisgI2VuZGlmICAgLyogVVNFX0lDVSAqLw0KICAJ -ew0KICAJCXRleHQJICAgKnN0cmluZyA9IFBHX0dFVEFSR19URVhUX1BfQ09Q -WSgwKTsNCiAgCQljaGFyCSAgICpwdHI7DQoqKioqKioqKioqKioqKioNCioq -KiAyMzIsMjM3ICoqKioNCi0tLSAzNDgsNDE4IC0tLS0NCiAgRGF0dW0NCiAg -dXBwZXIoUEdfRlVOQ1RJT05fQVJHUykNCiAgew0KKyAjaWZkZWYgVVNFX0lD -VQ0KKyAJLyogdXNlIElDVSBvbmx5IHdoZW4gbWF4IGVuY29kaW5nIGxlbmd0 -aCA+IG9uZSAqLw0KKyAJaWYgKHBnX2RhdGFiYXNlX2VuY29kaW5nX21heF9s -ZW5ndGgoKSA+IDEpDQorIAl7DQorIAkJdGV4dAkgICAqc3RyaW5nID0gUEdf -R0VUQVJHX1RFWFRfUCgwKTsNCisgCQl0ZXh0CSAgICpyZXN1bHQ7DQorIAkJ -VUNoYXIgICAgICAgc291cmNlYnVmW1NUQUNLQlVGTEVOXSwgZGVzdGJ1ZltT -VEFDS0JVRkxFTl07DQorIAkJVUNoYXIgICAgICAqc291cmNlLCAqZGVzdDsN -CisgCQlpbnQzMl90ICAgICBidWZsZW4sIGFyZ2xlbjsNCisgDQorIAkJVUVy -cm9yQ29kZSAgc3RhdHVzID0gVV9aRVJPX0VSUk9SOw0KKyANCisgCQlpZiAo -Y29udiA9PSBOVUxMKQ0KKyAJCXsNCisgCQkJY29udiA9IHVjbnZfb3BlbihO -VUxMLCAmc3RhdHVzKTsNCisgCQkJaWYgKFVfRkFJTFVSRShzdGF0dXMpKQ0K -KyAJCQl7DQorIAkJCQllcmVwb3J0KEVSUk9SLA0KKyAJCQkJCQkoZXJyY29k -ZShzdGF0dXMpLA0KKyAJCQkJCQkgZXJybXNnKCJJQ1UgZXJyb3I6IENvdWxk -IG5vdCBnZXQgY29udmVydGVyIGZvciBcIiVzXCIiLCB1Y252X2dldERlZmF1 -bHROYW1lKCkpKSk7DQorIAkJCX0NCisgCQl9DQorIAkJYXJnbGVuID0gVkFS -U0laRShzdHJpbmcpIC0gVkFSSERSU1o7DQorIAkJaWYgKGFyZ2xlbiAqIHNp -emVvZihVQ2hhcikgPj0gU1RBQ0tCVUZMRU4pDQorIAkJew0KKyAJCQlidWZs -ZW4gPSBhcmdsZW4gKiBzaXplb2YoVUNoYXIpICsgMTsNCisgIAkJCXNvdXJj -ZSA9IHBhbGxvYyhidWZsZW4pOw0KKyAJCQlkZXN0ID0gcGFsbG9jKGJ1Zmxl -bik7DQorIAkJfQ0KKyAJCWVsc2UNCisgCQl7DQorIAkJCWJ1ZmxlbiA9IFNU -QUNLQlVGTEVOOw0KKyAJCQlzb3VyY2UgPSBzb3VyY2VidWY7DQorIAkJCWRl -c3QgPSBkZXN0YnVmOw0KKyAJCX0NCisgCQkvLyBjb252ZXJ0IHRvIFVURi0x -Ng0KKyAJCXVjbnZfdG9VQ2hhcnMoY29udiwgc291cmNlLCBidWZsZW4sIFZB -UkRBVEEoc3RyaW5nKSwgYXJnbGVuLCAmc3RhdHVzKTsNCisgCQlpZiAoVV9G -QUlMVVJFKHN0YXR1cykpDQorIAkJew0KKyAJCQllcmVwb3J0KEVSUk9SLA0K -KyAJCQkJCQkoZXJyY29kZShzdGF0dXMpLA0KKyAJCQkJCQkgZXJybXNnKCJJ -Q1UgZXJyb3I6IENvdWxkIG5vdCBjb252ZXJ0IHN0cmluZyIpKSk7DQorIAkJ -fQ0KKyANCisgCQkvLyBydW4gZGVzaXJlZCBmdW5jdGlvbg0KKyAJCWJ1Zmxl -biA9IHVfc3RyVG9VcHBlcihkZXN0LCBidWZsZW4sIHNvdXJjZSwgLTEsIE5V -TEwsICZzdGF0dXMpOw0KKyAJCWlmIChVX0ZBSUxVUkUoc3RhdHVzKSkNCisg -CQl7DQorIAkJCWVyZXBvcnQoRVJST1IsDQorIAkJCQkJCShlcnJjb2RlKHN0 -YXR1cyksDQorIAkJCQkJCSBlcnJtc2coIklDVSBlcnJvcjogQ291bGQgbm90 -IG1vZGlmeSBjYXNlIikpKTsNCisgCQl9DQorIA0KKyAJCS8vIGFuZCBjb252 -ZXJ0IG1vZGlmaWVkIHV0Zi0xNiBzdHJpbmcgYmFjayB0byB0ZXh0DQorIAkJ -cmVzdWx0ID0gVUNoYXJ0b3RleHQoZGVzdCwgYnVmbGVuKTsNCisgDQorIAkJ -aWYgKGFyZ2xlbiAqIHNpemVvZihVQ2hhcikgPj0gU1RBQ0tCVUZMRU4pDQor -IAkJew0KKyAJCQlwZnJlZShzb3VyY2UpOw0KKyAJCQlwZnJlZShkZXN0KTsN -CisgCQl9DQorIAkJUEdfUkVUVVJOX1RFWFRfUChyZXN1bHQpOw0KKyAJfQ0K -KyAJZWxzZQ0KKyAjZWxzZQ0KICAjaWZkZWYgVVNFX1dJREVfVVBQRVJfTE9X -RVINCiAgCS8qDQogIAkgKglVc2Ugd2lkZSBjaGFyIGNvZGUgb25seSB3aGVu -IG1heCBlbmNvZGluZyBsZW5ndGggPiAxIGFuZCBjdHlwZSAhPSBDLg0KKioq -KioqKioqKioqKioqDQoqKiogMjU4LDI2MyAqKioqDQotLS0gNDM5LDQ0NSAt -LS0tDQogIAl9DQogIAllbHNlDQogICNlbmRpZiAgIC8qIFVTRV9XSURFX1VQ -UEVSX0xPV0VSICovDQorICNlbmRpZiAgIC8qIFVTRV9JQ1UgKi8NCiAgCXsN -CiAgCQl0ZXh0CSAgICpzdHJpbmcgPSBQR19HRVRBUkdfVEVYVF9QX0NPUFko -MCk7DQogIAkJY2hhcgkgICAqcHRyOw0KKioqKioqKioqKioqKioqDQoqKiog -MzAxLDMwNiAqKioqDQotLS0gNDgzLDU1MyAtLS0tDQogIERhdHVtDQogIGlu -aXRjYXAoUEdfRlVOQ1RJT05fQVJHUykNCiAgew0KKyAjaWZkZWYgVVNFX0lD -VQ0KKyAJLyogdXNlIElDVSBvbmx5IHdoZW4gbWF4IGVuY29kaW5nIGxlbmd0 -aCA+IG9uZSAqLw0KKyAJaWYgKHBnX2RhdGFiYXNlX2VuY29kaW5nX21heF9s -ZW5ndGgoKSA+IDEpDQorIAl7DQorIAkJdGV4dAkgICAqc3RyaW5nID0gUEdf -R0VUQVJHX1RFWFRfUCgwKTsNCisgCQl0ZXh0CSAgICpyZXN1bHQ7DQorIAkJ -VUNoYXIgICAgICAgc291cmNlYnVmW1NUQUNLQlVGTEVOXSwgZGVzdGJ1ZltT -VEFDS0JVRkxFTl07DQorIAkJVUNoYXIgICAgICAqc291cmNlLCAqZGVzdDsN -CisgCQlpbnQzMl90ICAgICBidWZsZW4sIGFyZ2xlbjsNCisgDQorIAkJVUVy -cm9yQ29kZSAgc3RhdHVzID0gVV9aRVJPX0VSUk9SOw0KKyANCisgCQlpZiAo -Y29udiA9PSBOVUxMKQ0KKyAJCXsNCisgCQkJY29udiA9IHVjbnZfb3BlbihO -VUxMLCAmc3RhdHVzKTsNCisgCQkJaWYgKFVfRkFJTFVSRShzdGF0dXMpKQ0K -KyAJCQl7DQorIAkJCQllcmVwb3J0KEVSUk9SLA0KKyAJCQkJCQkoZXJyY29k -ZShzdGF0dXMpLA0KKyAJCQkJCQkgZXJybXNnKCJJQ1UgZXJyb3I6IENvdWxk -IG5vdCBnZXQgY29udmVydGVyIGZvciBcIiVzXCIiLCB1Y252X2dldERlZmF1 -bHROYW1lKCkpKSk7DQorIAkJCX0NCisgCQl9DQorIAkJYXJnbGVuID0gVkFS -U0laRShzdHJpbmcpIC0gVkFSSERSU1o7DQorIAkJaWYgKGFyZ2xlbiAqIHNp -emVvZihVQ2hhcikgPj0gU1RBQ0tCVUZMRU4pDQorIAkJew0KKyAJCQlidWZs -ZW4gPSBhcmdsZW4gKiBzaXplb2YoVUNoYXIpICsgMTsNCisgIAkJCXNvdXJj -ZSA9IHBhbGxvYyhidWZsZW4pOw0KKyAJCQlkZXN0ID0gcGFsbG9jKGJ1Zmxl -bik7DQorIAkJfQ0KKyAJCWVsc2UNCisgCQl7DQorIAkJCWJ1ZmxlbiA9IFNU -QUNLQlVGTEVOOw0KKyAJCQlzb3VyY2UgPSBzb3VyY2VidWY7DQorIAkJCWRl -c3QgPSBkZXN0YnVmOw0KKyAJCX0NCisgCQkvLyBjb252ZXJ0IHRvIFVURi0x -Ng0KKyAJCXVjbnZfdG9VQ2hhcnMoY29udiwgc291cmNlLCBidWZsZW4sIFZB -UkRBVEEoc3RyaW5nKSwgYXJnbGVuLCAmc3RhdHVzKTsNCisgCQlpZiAoVV9G -QUlMVVJFKHN0YXR1cykpDQorIAkJew0KKyAJCQllcmVwb3J0KEVSUk9SLA0K -KyAJCQkJCQkoZXJyY29kZShzdGF0dXMpLA0KKyAJCQkJCQkgZXJybXNnKCJJ -Q1UgZXJyb3I6IENvdWxkIG5vdCBjb252ZXJ0IHN0cmluZyIpKSk7DQorIAkJ -fQ0KKyANCisgCQkvLyBydW4gZGVzaXJlZCBmdW5jdGlvbg0KKyAJCWJ1Zmxl -biA9IHVfc3RyVG9UaXRsZShkZXN0LCBidWZsZW4sIHNvdXJjZSwgLTEsIE5V -TEwsIE5VTEwsICZzdGF0dXMpOw0KKyAJCWlmIChVX0ZBSUxVUkUoc3RhdHVz -KSkNCisgCQl7DQorIAkJCWVyZXBvcnQoRVJST1IsDQorIAkJCQkJCShlcnJj -b2RlKHN0YXR1cyksDQorIAkJCQkJCSBlcnJtc2coIklDVSBlcnJvcjogQ291 -bGQgbm90IG1vZGlmeSBjYXNlIikpKTsNCisgCQl9DQorIA0KKyAJCS8vIGFu -ZCBjb252ZXJ0IG1vZGlmaWVkIHV0Zi0xNiBzdHJpbmcgYmFjayB0byB0ZXh0 -DQorIAkJcmVzdWx0ID0gVUNoYXJ0b3RleHQoZGVzdCwgYnVmbGVuKTsNCisg -DQorIAkJaWYgKGFyZ2xlbiAqIHNpemVvZihVQ2hhcikgPj0gU1RBQ0tCVUZM -RU4pDQorIAkJew0KKyAJCQlwZnJlZShzb3VyY2UpOw0KKyAJCQlwZnJlZShk -ZXN0KTsNCisgCQl9DQorIAkJUEdfUkVUVVJOX1RFWFRfUChyZXN1bHQpOw0K -KyAJfQ0KKyAJZWxzZQ0KKyAjZWxzZQ0KICAjaWZkZWYgVVNFX1dJREVfVVBQ -RVJfTE9XRVINCiAgCS8qDQogIAkgKglVc2Ugd2lkZSBjaGFyIGNvZGUgb25s -eSB3aGVuIG1heCBlbmNvZGluZyBsZW5ndGggPiAxIGFuZCBjdHlwZSAhPSBD -Lg0KKioqKioqKioqKioqKioqDQoqKiogMzM0LDMzOSAqKioqDQotLS0gNTgx -LDU4NyAtLS0tDQogIAl9DQogIAllbHNlDQogICNlbmRpZiAgIC8qIFVTRV9X -SURFX1VQUEVSX0xPV0VSICovDQorICNlbmRpZiAgIC8qIFVTRV9JQ1UgKi8N -CiAgCXsNCiAgCQl0ZXh0CSAgICpzdHJpbmcgPSBQR19HRVRBUkdfVEVYVF9Q -X0NPUFkoMCk7DQogIAkJaW50CQkJd2FzYWxudW0gPSAwOw0KSW5kZXg6IHNy -Yy9iYWNrZW5kL3V0aWxzL2FkdC92YXJsZW5hLmMNCj09PT09PT09PT09PT09 -PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 -PT09PT09PT0NClJDUyBmaWxlOiAvcHJvamVjdHMvY3Zzcm9vdC9wZ3NxbC9z -cmMvYmFja2VuZC91dGlscy9hZHQvdmFybGVuYS5jLHYNCnJldHJpZXZpbmcg -cmV2aXNpb24gMS4xMzANCmRpZmYgLWMgLXIxLjEzMCB2YXJsZW5hLmMNCioq -KiBzcmMvYmFja2VuZC91dGlscy9hZHQvdmFybGVuYS5jCTI5IEp1bCAyMDA1 -IDAzOjE3OjU1IC0wMDAwCTEuMTMwDQotLS0gc3JjL2JhY2tlbmQvdXRpbHMv -YWR0L3ZhcmxlbmEuYwkzMCBKdWwgMjAwNSAxNzo0NjozNCAtMDAwMA0KKioq -KioqKioqKioqKioqDQoqKiogMzAsMzUgKioqKg0KLS0tIDMwLDQxIC0tLS0N -CiAgI2luY2x1ZGUgInV0aWxzL3BnX2xvY2FsZS5oIg0KICAjaW5jbHVkZSAi -cmVnZXgvcmVnZXguaCINCiAgDQorICNpZmRlZiBVU0VfSUNVDQorICNpbmNs -dWRlIDx1bmljb2RlL3V0eXBlcy5oPiAgIC8qIEJhc2ljIElDVSBkYXRhIHR5 -cGVzICovDQorICNpbmNsdWRlIDx1bmljb2RlL3VjbnYuaD4gICAgIC8qIEMg -ICBDb252ZXJ0ZXIgQVBJICAgICovDQorICNpbmNsdWRlIDx1bmljb2RlL3Vj -b2wuaD4NCisgI2luY2x1ZGUgPHVuaWNvZGUvdWxvYy5oPg0KKyAjZW5kaWYg -LyogVVNFX0lDVSAqLw0KICANCiAgdHlwZWRlZiBzdHJ1Y3QgdmFybGVuYSB1 -bmtub3duOw0KICANCioqKioqKioqKioqKioqKg0KKioqIDg0NCw4NzQgKioq -Kg0KICANCiAgCWlmICghbGNfY29sbGF0ZV9pc19jKCkpDQogIAl7DQohIAkJ -Y2hhcgkJYTFidWZbU1RBQ0tCVUZMRU5dOw0KISAJCWNoYXIJCWEyYnVmW1NU -QUNLQlVGTEVOXTsNCiEgCQljaGFyCSAgICphMXAsDQohIAkJCQkgICAqYTJw -Ow0KICANCiEgCQlpZiAobGVuMSA+PSBTVEFDS0JVRkxFTikNCiEgCQkJYTFw -ID0gKGNoYXIgKikgcGFsbG9jKGxlbjEgKyAxKTsNCiEgCQllbHNlDQohIAkJ -CWExcCA9IGExYnVmOw0KISAJCWlmIChsZW4yID49IFNUQUNLQlVGTEVOKQ0K -ISAJCQlhMnAgPSAoY2hhciAqKSBwYWxsb2MobGVuMiArIDEpOw0KICAJCWVs -c2UNCiEgCQkJYTJwID0gYTJidWY7DQogIA0KISAJCW1lbWNweShhMXAsIGFy -ZzEsIGxlbjEpOw0KISAJCWExcFtsZW4xXSA9ICdcMCc7DQohIAkJbWVtY3B5 -KGEycCwgYXJnMiwgbGVuMik7DQohIAkJYTJwW2xlbjJdID0gJ1wwJzsNCiEg -DQohIAkJcmVzdWx0ID0gc3RyY29sbChhMXAsIGEycCk7DQohIA0KISAJCWlm -IChsZW4xID49IFNUQUNLQlVGTEVOKQ0KISAJCQlwZnJlZShhMXApOw0KISAJ -CWlmIChsZW4yID49IFNUQUNLQlVGTEVOKQ0KISAJCQlwZnJlZShhMnApOw0K -ICAJfQ0KICAJZWxzZQ0KICAJew0KLS0tIDg1MCw5ODYgLS0tLQ0KICANCiAg -CWlmICghbGNfY29sbGF0ZV9pc19jKCkpDQogIAl7DQohICNpZmRlZiBVU0Vf -SUNVDQohICNkZWZpbmUgVVNUQUNLQlVGTEVOCQlTVEFDS0JVRkxFTiAvIHNp -emVvZihVQ2hhcikNCiEgCQlpZiAocGdfZGF0YWJhc2VfZW5jb2RpbmdfbWF4 -X2xlbmd0aCgpID4gMSkNCiEgCQl7DQohIAkJCVVDaGFyCQlhMWJ1ZltVU1RB -Q0tCVUZMRU5dLA0KISAJCSAgICAgICAgIAkJYTJidWZbVVNUQUNLQlVGTEVO -XTsNCiEgCQkJaW50CQkJYTFsZW4gPSBVU1RBQ0tCVUZMRU4sDQohIAkJCSAg -CQkJYTJsZW4gPSBVU1RBQ0tCVUZMRU47CQ0KISAJCQlVQ2hhcgkgICAqYTFw -LA0KISAJCQkgICAgICAgICAgICphMnA7DQohIA0KISAJCQlzdGF0aWMgVUNv -bGxhdG9yICogY29sbGF0b3IgPSBOVUxMOw0KISAJCQlzdGF0aWMgVUNvbnZl -cnRlciAqIGNvbnYgPSBOVUxMOw0KISAJCQlVRXJyb3JDb2RlICBzdGF0dXMg -PSBVX1pFUk9fRVJST1I7DQogIA0KISAJCQlpZiAoY29udiA9PSBOVUxMKQ0K -ISAJCQl7DQohIAkJCQljb252ID0gdWNudl9vcGVuKE5VTEwsICZzdGF0dXMp -Ow0KISAJCQkJaWYgKFVfRkFJTFVSRShzdGF0dXMpIHx8IGNvbnYgPT0gTlVM -TCkNCiEgCQkJCXsNCiEgCQkJCQllcmVwb3J0KEVSUk9SLA0KISAJCQkJCQkJ -KGVycmNvZGUoc3RhdHVzKSwNCiEgCQkJCQkJCSBlcnJtc2coIklDVSBlcnJv -cjogdmFybGVuYS5jLCBjb3VsZCBub3QgZ2V0IGNvbnZlcnRlciBmb3IgXCIl -c1wiIiwgdWNudl9nZXREZWZhdWx0TmFtZSgpKSkpOw0KISAJCQkJfQ0KISAJ -CQl9DQohIA0KISAJCQkvKiBXZSBrZWVwIGEgc3RhdGljIGNvbGxhdG9yICJm -b3JldmVyIiwgc2luY2UgaXQgaXMgaGFyZA0KISAJCQkgKiBjb2RlZCBpbnRv -IHRoZSBkYXRhYmFzZSBjbHVzdGVyIGF0IGluaXRkYiB0aW1lDQohIAkJCSAq -IGFueXdheS4gQ3JlYXRlIGl0IGZpcnN0IHRpbWUgd2UgZ2V0IGhlcmUuICov -DQohIAkJCWlmIChjb2xsYXRvciA9PSBOVUxMKQ0KISAJCQl7DQohIAkJCQkv -KiBFeHBlY3QgTENfQ09MTEFURSB0byBiZSBzZXQgdG8gc29tZXRoaW5nIHRo -YXQgSUNVDQohIAkJCQkgKiB3aWxsIHVuZGVyc3RhbmQuIFRoaXMgaXMgcXVp -dGUgcHJvYmFibGUsIHNpbmNlIElDVQ0KISAJCQkJICogZG9lcyBhIGxvdCBv -ZiBoZXVyaXN0aWNzIHdpdGggdGhpcyBhcmd1bWVudC4gSSdkDQohIAkJCQkg -KiByYXRoZXIgc2V0IHRoaXMgaW4geGxvZy5jLCBidXQgaXQgc2VlbXMgSUNV -IGZvcmdldHMNCiEgCQkJCSAqIGl0Pz8/ICovDQohICNpZm5kZWYgV0lOMzIN -CiEgCQkJCXVsb2Nfc2V0RGVmYXVsdChzZXRsb2NhbGUoTENfQ09MTEFURSwg -TlVMTCksICZzdGF0dXMpOw0KISAjZWxzZQ0KISAJCQkJLyogV2luMzIgbG9j -YWxlIG5hbWVzIGFyZSBjb21wbGV0ZWx5IGRpZmZlcmVudCBmcm9tIHdoYXQg -SUNVIGV4cGVjdHMsIHNvIA0KISAJCQkJICAgd2UgbmVlZCB0byBkbyBzb21l -IGNvbnZlcnNpb24gKi8NCiEgCQkJCXVsb2Nfc2V0RGVmYXVsdChwZ3dpbjMy -X2xvY2FsZW1hcChzZXRsb2NhbGUoTENfQ09MTEFURSwgTlVMTCkpLCAmc3Rh -dHVzKTsNCiEgI2VuZGlmDQohIAkJCQlpZihVX0ZBSUxVUkUoc3RhdHVzKSkN -CiEgCQkJCXsNCiEgCQkJCQllcmVwb3J0KFdBUk5JTkcsDQohIAkJCQkJCQko -ZXJyY29kZShzdGF0dXMpLA0KISAJCQkJCQkJIGVycm1zZygiSUNVIEVycm9y -OiB2YXJsZW5hLmMsIGNvdWxkIG5vdCBzZXQgZGVmYXVsdCBsY19jb2xsYXRl -IikpKTsNCiEgCQkJCX0NCiEgCQkJCWNvbGxhdG9yID0gdWNvbF9vcGVuKE5V -TEwsICZzdGF0dXMpOw0KISAJCQkJaWYgKFVfRkFJTFVSRShzdGF0dXMpKQ0K -ISAJCQkJew0KISAJCQkJCWVyZXBvcnQoV0FSTklORywNCiEgCQkJCQkJCShl -cnJjb2RlKHN0YXR1cyksDQohIAkJCQkJCQkgZXJybXNnKCJJQ1UgRXJyb3I6 -IHZhcmxlbmEuYywgY291bGQgbm90IG9wZW4gY29sbGF0b3IiKSkpOw0KISAJ -CQkJfQ0KISAJCQl9DQohIAkJCQ0KISAJCQlpZiAobGVuMSA+PSBVU1RBQ0tC -VUZMRU4gLyBzaXplb2YoVUNoYXIpKQ0KISAJCQl7DQohIAkJCQlhMWxlbiA9 -IGxlbjEgKiBzaXplb2YoVUNoYXIpICsgMjsNCiEgCQkJCWExcCA9IChVQ2hh -ciAqKSBwYWxsb2MoYTFsZW4pOwkNCiEgCQkJfQ0KISAJCQllbHNlDQohIAkJ -CQlhMXAgPSBhMWJ1ZjsNCiEgDQohIAkJCWlmIChsZW4yID49IFVTVEFDS0JV -RkxFTiAvIHNpemVvZihVQ2hhcikpDQohIAkJCXsNCiEgCQkJCWEybGVuID0g -bGVuMiAqIHNpemVvZihVQ2hhcikgKyAyOw0KISAJCQkJYTJwID0gKFVDaGFy -ICopIHBhbGxvYyhhMmxlbik7DQohIAkJCX0NCiEgCQkJZWxzZQ0KISAJCQkJ -YTJwID0gYTJidWY7DQohIA0KISAJCQl1Y252X3RvVUNoYXJzKGNvbnYsIGEx -cCwgYTFsZW4sIGFyZzEsIGxlbjEsICZzdGF0dXMpOw0KISAJCQlpZihVX0ZB -SUxVUkUoc3RhdHVzKSkNCiEgCQkJew0KISAJCQkJZXJlcG9ydChXQVJOSU5H -LA0KISAJCQkJCQkoZXJyY29kZShzdGF0dXMpLA0KISAJCQkJCQkgZXJybXNn -KCJJQ1UgRXJyb3I6IHZhcmxlbmEuYywgY291bGQgbm90IGNvbnZlcnQgdG8g -VUNoYXJzIikpKTsNCiEgCQkJfQ0KISAJCQl1Y252X3RvVUNoYXJzKGNvbnYs -IGEycCwgYTJsZW4sIGFyZzIsIGxlbjIsICZzdGF0dXMpOw0KISAJCQlpZihV -X0ZBSUxVUkUoc3RhdHVzKSkNCiEgCQkJew0KISAJCQkJZXJlcG9ydChXQVJO -SU5HLA0KISAJCQkJCQkoZXJyY29kZShzdGF0dXMpLA0KISAJCQkJCQkgZXJy -bXNnKCJJQ1UgRXJyb3I6IHZhcmxlbmEuYywgY291bGQgbm90IGNvbnZlcnQg -dG8gVUNoYXJzIikpKTsNCiEgCQkJfQ0KISANCiEgCQkJcmVzdWx0ID0gdWNv -bF9zdHJjb2xsKGNvbGxhdG9yLCBhMXAsIC0xLCBhMnAsIC0xKTsNCiEgCQkJ -aWYoVV9GQUlMVVJFKHN0YXR1cykpDQohIAkJCXsNCiEgCQkJCWVyZXBvcnQo -V0FSTklORywNCiEgCQkJCQkJKGVycmNvZGUoc3RhdHVzKSwNCiEgCQkJCQkJ -IGVycm1zZygiSUNVIEVycm9yOiB2YXJsZW5hLmMsIGNvdWxkIG5vdCBjb2xs -YXRlIikpKTsNCiEgCQkJfQ0KISANCiEgCQkJaWYgKGxlbjEgKiBzaXplb2Yo -VUNoYXIpID49IFVTVEFDS0JVRkxFTikNCiEgCQkJCXBmcmVlKGExcCk7DQoh -IAkJCWlmIChsZW4yICogc2l6ZW9mKFVDaGFyKSA+PSBVU1RBQ0tCVUZMRU4p -DQohIAkJCQlwZnJlZShhMnApOw0KISAJCX0NCiAgCQllbHNlDQohICNlbmRp -ZiAvKiBVU0VfSUNVICovDQohIAkJew0KISAJCQljaGFyCQlhMWJ1ZltTVEFD -S0JVRkxFTl07DQohIAkJCWNoYXIJCWEyYnVmW1NUQUNLQlVGTEVOXTsNCiEg -CQkJY2hhcgkgICAqYTFwLA0KISAJCQkJCSAgICphMnA7DQogIA0KISAJCQlp -ZiAobGVuMSA+PSBTVEFDS0JVRkxFTikNCiEgCQkJCWExcCA9IChjaGFyICop -IHBhbGxvYyhsZW4xICsgMSk7DQohIAkJCWVsc2UNCiEgCQkJCWExcCA9IGEx -YnVmOw0KISAJCQlpZiAobGVuMiA+PSBTVEFDS0JVRkxFTikNCiEgCQkJCWEy -cCA9IChjaGFyICopIHBhbGxvYyhsZW4yICsgMSk7DQohIAkJCWVsc2UNCiEg -CQkJCWEycCA9IGEyYnVmOw0KISAJCQkNCiEgCQkJbWVtY3B5KGExcCwgYXJn -MSwgbGVuMSk7DQohIAkJCWExcFtsZW4xXSA9ICdcMCc7DQohIAkJCW1lbWNw -eShhMnAsIGFyZzIsIGxlbjIpOw0KISAJCQlhMnBbbGVuMl0gPSAnXDAnOw0K -ISAJCQkNCiEgCQkJcmVzdWx0ID0gc3RyY29sbChhMXAsIGEycCk7DQohIAkJ -CQ0KISAJCQlpZiAobGVuMSA+PSBTVEFDS0JVRkxFTikNCiEgCQkJCXBmcmVl -KGExcCk7DQohIAkJCWlmIChsZW4yID49IFNUQUNLQlVGTEVOKQ0KISAJCQkJ -cGZyZWUoYTJwKTsNCiEgCQl9DQogIAl9DQogIAllbHNlDQogIAl7DQpJbmRl -eDogc3JjL2JhY2tlbmQvdXRpbHMvbWIvZW5jbmFtZXMuYw0KPT09PT09PT09 -PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 -PT09PT09PT09PT09PQ0KUkNTIGZpbGU6IC9wcm9qZWN0cy9jdnNyb290L3Bn -c3FsL3NyYy9iYWNrZW5kL3V0aWxzL21iL2VuY25hbWVzLmMsdg0KcmV0cmll -dmluZyByZXZpc2lvbiAxLjI1DQpkaWZmIC1jIC1yMS4yNSBlbmNuYW1lcy5j -DQoqKiogc3JjL2JhY2tlbmQvdXRpbHMvbWIvZW5jbmFtZXMuYwkxNCBNYXIg -MjAwNSAxODozMToyMCAtMDAwMAkxLjI1DQotLS0gc3JjL2JhY2tlbmQvdXRp -bHMvbWIvZW5jbmFtZXMuYwkyOCBKdWwgMjAwNSAxNzoxNjo1MiAtMDAwMA0K -KioqKioqKioqKioqKioqDQoqKiogMzc1LDM4MCAqKioqDQotLS0gMzc1LDQ5 -NiAtLS0tDQogIAl9DQogIH07DQogIA0KKyAjaWZkZWYgVVNFX0lDVQ0KKyAv -Kg0KKyAgKiBUcnkgdG8gbWFwIG1vc3QgaW50ZXJuYWwgY2hhcmFjdGVyIGVu -Y29kaW5ncyB0byB0aGUgcHJvcGVyIGFuZA0KKyAgKiBwcmVmZXJyZWQgSUFO -QSBzdHJpbmcuIFVzZSB0aGlzIGluIG1idXRpbHMuYyB0byBmZWVkIElDVSBp -bmZvIGFib3V0DQorICAqIHRoZSBkYXRhYmFzZSdzIGNoYXJhY3RlciBlbmNv -ZGluZy4NCisgICoNCisgICogUGFsbGUgR2lyZ2Vuc29obiwgMjAwNQ0KKyAg -Ki8NCisgDQorIHBnX2VuYzJuYW1lIHBnX2VuYzJpYW5hbmFtZV90YmxbXSA9 -DQorIHsNCisgCXsNCisgCQkiVVMtQVNDSUkiLCBQR19TUUxfQVNDSUkNCisg -CX0sDQorIAl7DQorIAkJIkVVQy1KUCIsIFBHX0VVQ19KUA0KKyAJfSwNCisg -CXsNCisgCQkiR0IyMzEyIiwgUEdfRVVDX0NODQorIAl9LA0KKyAJew0KKyAJ -CSJFVUMtS1IiLCBQR19FVUNfS1INCisgCX0sDQorIAl7DQorIAkJIklTTy0y -MDIyLUNOIiwgUEdfRVVDX1RXDQorIAl9LA0KKyAJew0KKyAJCSJLU19DXzU2 -MDEtMTk4NyIsIFBHX0pPSEFCICAvKiBlaXRoZXIgS1NfQ181NjAxLTE5ODcg -b3IgSVNPLTIwMjItS1IgPz8/ICovDQorIAl9LA0KKyAJew0KKyAJCSJVVEYt -OCIsIFBHX1VURjgNCisgCX0sDQorIAl7DQorIAkJIk1VTEVfSU5URVJOQUwi -LCBQR19NVUxFX0lOVEVSTkFMICAvKiBpcyBub3QgZm9yIHJlYWwgKi8NCisg -CX0sDQorIAl7DQorIAkJIklTTy04ODU5LTEiLCBQR19MQVRJTjENCisgCX0s -DQorIAl7DQorIAkJIklTTy04ODU5LTIiLCBQR19MQVRJTjINCisgCX0sDQor -IAl7DQorIAkJIklTTy04ODU5LTMiLCBQR19MQVRJTjMNCisgCX0sDQorIAl7 -DQorIAkJIklTTy04ODU5LTQiLCBQR19MQVRJTjQNCisgCX0sDQorIAl7DQor -IAkJIklTTy04ODU5LTkiLCBQR19MQVRJTjUNCisgCX0sDQorIAl7DQorIAkJ -IklTTy04ODU5LTEwIiwgUEdfTEFUSU42DQorIAl9LA0KKyAJew0KKyAJCSJJ -U08tODg1OS0xMyIsIFBHX0xBVElONw0KKyAJfSwNCisgCXsNCisgCQkiSVNP -LTg4NTktMTQiLCBQR19MQVRJTjgNCisgCX0sDQorIAl7DQorIAkJIklTTy04 -ODU5LTE1IiwgUEdfTEFUSU45DQorIAl9LA0KKyAJew0KKyAJCSJJU08tODg1 -OS0xNiIsIFBHX0xBVElOMTANCisgCX0sDQorIAl7DQorIAkJIndpbmRvd3Mt -MTI1NiIsIFBHX1dJTjEyNTYNCisgCX0sDQorIAl7DQorIAkJIndpbmRvd3Mt -MTI1OCIsIFBHX1dJTjEyNTgNCisgCX0sDQorIAl7DQorIAkJIndpbmRvd3Mt -ODc0IiwgUEdfV0lOODc0DQorIAl9LA0KKyAJew0KKyAJCSJLT0k4LVIiLCBQ -R19LT0k4Ug0KKyAJfSwNCisgCXsNCisgCQkid2luZG93cy0xMjUxIiwgUEdf -V0lOMTI1MQ0KKyAJfSwNCisgCXsNCisgCQkiSUJNODY2IiwgUEdfV0lOODY2 -DQorIAl9LA0KKyAJew0KKyAJCSJJU08tODg1OS01IiwgUEdfSVNPXzg4NTlf -NQ0KKyAJfSwNCisgCXsNCisgCQkiSVNPLTg4NTktNiIsIFBHX0lTT184ODU5 -XzYNCisgCX0sDQorIAl7DQorIAkJIklTTy04ODU5LTciLCBQR19JU09fODg1 -OV83DQorIAl9LA0KKyAJew0KKyAJCSJJU08tODg1OS04IiwgUEdfSVNPXzg4 -NTlfOA0KKyAJfSwNCisgCXsNCisgCQkid2luZG93cy0xMjUwIiwgUEdfV0lO -MTI1MA0KKyAJfSwNCisgCXsNCisgCQkiU2hpZnRfSklTIiwgUEdfU0pJUw0K -KyAJfSwNCisgCXsNCisgCQkiQmlnNSIsIFBHX0JJRzUNCisgCX0sDQorIAl7 -DQorIAkJIkdCSyIsIFBHX0dCSw0KKyAJfSwNCisgCXsNCisgCQkiY3A5NDki -LCBQR19VSEMNCisgCX0sDQorIAl7DQorIAkJIkdCMTgwMzAiLCBQR19HQjE4 -MDMwDQorIAl9DQorIH07DQorICNlbmRpZiAvKiBVU0VfSUNVICovDQorIA0K -ICAvKiAtLS0tLS0tLS0tDQogICAqIEVuY29kaW5nIGNoZWNrcywgZm9yIGVy -cm9yIHJldHVybnMgLTEgZWxzZSBlbmNvZGluZyBpZA0KICAgKiAtLS0tLS0t -LS0tDQpJbmRleDogc3JjL2JhY2tlbmQvdXRpbHMvbWIvbWJ1dGlscy5jDQo9 -PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 -PT09PT09PT09PT09PT09PT09PT09DQpSQ1MgZmlsZTogL3Byb2plY3RzL2N2 -c3Jvb3QvcGdzcWwvc3JjL2JhY2tlbmQvdXRpbHMvbWIvbWJ1dGlscy5jLHYN -CnJldHJpZXZpbmcgcmV2aXNpb24gMS41MA0KZGlmZiAtYyAtcjEuNTAgbWJ1 -dGlscy5jDQoqKiogc3JjL2JhY2tlbmQvdXRpbHMvbWIvbWJ1dGlscy5jCTEw -IEp1bCAyMDA1IDIxOjEzOjU5IC0wMDAwCTEuNTANCi0tLSBzcmMvYmFja2Vu -ZC91dGlscy9tYi9tYnV0aWxzLmMJMjggSnVsIDIwMDUgMTY6MzY6NTQgLTAw -MDANCioqKioqKioqKioqKioqKg0KKioqIDE1LDIwICoqKioNCi0tLSAxNSwy -MyAtLS0tDQogICNpbmNsdWRlICJ1dGlscy9tZW11dGlscy5oIg0KICAjaW5j -bHVkZSAidXRpbHMvc3lzY2FjaGUuaCINCiAgI2luY2x1ZGUgImNhdGFsb2cv -bmFtZXNwYWNlLmgiDQorICNpZmRlZiBVU0VfSUNVDQorICNpbmNsdWRlIDx1 -bmljb2RlL3VjbnYuaD4NCisgI2VuZGlmIC8qIFVTRV9JQ1UgKi8NCiAgDQog -IC8qDQogICAqIFdlIGhhbmRsZSBmb3IgYWN0dWFsIEZFIGFuZCBCRSBlbmNv -ZGluZyBzZXR0aW5nIGVuY29kaW5nLWlkZW50aWZpY2F0b3INCioqKioqKioq -KioqKioqKg0KKioqIDU3Niw1ODEgKioqKg0KLS0tIDU3OSw1ODcgLS0tLQ0K -ICANCiAgCURhdGFiYXNlRW5jb2RpbmcgPSAmcGdfZW5jMm5hbWVfdGJsW2Vu -Y29kaW5nXTsNCiAgCUFzc2VydChEYXRhYmFzZUVuY29kaW5nLT5lbmNvZGlu -ZyA9PSBlbmNvZGluZyk7DQorICNpZmRlZiBVU0VfSUNVDQorIAl1Y252X3Nl -dERlZmF1bHROYW1lKCgmcGdfZW5jMmlhbmFuYW1lX3RibFtlbmNvZGluZ10p -LT5uYW1lKTsNCisgI2VuZGlmDQogIH0NCiAgDQogIHZvaWQNCkluZGV4OiBz -cmMvaW5jbHVkZS9wZ19jb25maWcuaC5pbg0KPT09PT09PT09PT09PT09PT09 -PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 -PT09PQ0KUkNTIGZpbGU6IC9wcm9qZWN0cy9jdnNyb290L3Bnc3FsL3NyYy9p -bmNsdWRlL3BnX2NvbmZpZy5oLmluLHYNCnJldHJpZXZpbmcgcmV2aXNpb24g -MS44Ng0KZGlmZiAtYyAtcjEuODYgcGdfY29uZmlnLmguaW4NCioqKiBzcmMv -aW5jbHVkZS9wZ19jb25maWcuaC5pbgkxIEp1bCAyMDA1IDE4OjE3OjMxIC0w -MDAwCTEuODYNCi0tLSBzcmMvaW5jbHVkZS9wZ19jb25maWcuaC5pbgkyOCBK -dWwgMjAwNSAxNjozNjo1NCAtMDAwMA0KKioqKioqKioqKioqKioqDQoqKiog -NjQyLDY0NyAqKioqDQotLS0gNjQyLDY1MCAtLS0tDQogIC8qIERlZmluZSB0 -byBidWlsZCB3aXRoIChPcGVuKVNTTCBzdXBwb3J0LiAoLS13aXRoLW9wZW5z -c2wpICovDQogICN1bmRlZiBVU0VfU1NMDQogIA0KKyAvKiBEZWZpbmUgdG8g -YnVpbGQgd2l0aCBJQ1Ugc3VwcG9ydC4gKC0td2l0aC1pY3UpICovDQorICN1 -bmRlZiBVU0VfSUNVDQorIA0KICAvKiBEZWZpbmUgdG8gc2VsZWN0IFN5c1Yt -c3R5bGUgc2VtYXBob3Jlcy4gKi8NCiAgI3VuZGVmIFVTRV9TWVNWX1NFTUFQ -SE9SRVMNCiAgDQpJbmRleDogc3JjL2luY2x1ZGUvbWIvcGdfd2NoYXIuaA0K -PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 -PT09PT09PT09PT09PT09PT09PT09PQ0KUkNTIGZpbGU6IC9wcm9qZWN0cy9j -dnNyb290L3Bnc3FsL3NyYy9pbmNsdWRlL21iL3BnX3djaGFyLmgsdg0KcmV0 -cmlldmluZyByZXZpc2lvbiAxLjU5DQpkaWZmIC1jIC1yMS41OSBwZ193Y2hh -ci5oDQoqKiogc3JjL2luY2x1ZGUvbWIvcGdfd2NoYXIuaAkxNSBKdW4gMjAw -NSAwMDoxNTowOCAtMDAwMAkxLjU5DQotLS0gc3JjL2luY2x1ZGUvbWIvcGdf -d2NoYXIuaAkyOCBKdWwgMjAwNSAxNjozNjo1NCAtMDAwMA0KKioqKioqKioq -KioqKioqDQoqKiogMjM1LDI0MCAqKioqDQotLS0gMjM1LDI0MSAtLS0tDQog -IH0gcGdfZW5jMm5hbWU7DQogIA0KICBleHRlcm4gcGdfZW5jMm5hbWUgcGdf -ZW5jMm5hbWVfdGJsW107DQorIGV4dGVybiBwZ19lbmMybmFtZSBwZ19lbmMy -aWFuYW5hbWVfdGJsW107DQogIA0KICBleHRlcm4gcGdfZW5jbmFtZSAqcGdf -Y2hhcl90b19lbmNuYW1lX3N0cnVjdChjb25zdCBjaGFyICpuYW1lKTsNCiAg -DQpJbmRleDogc3JjL2luY2x1ZGUvcG9ydC93aW4zMi5oDQo9PT09PT09PT09 -PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 -PT09PT09PT09PT09DQpSQ1MgZmlsZTogL3Byb2plY3RzL2N2c3Jvb3QvcGdz -cWwvc3JjL2luY2x1ZGUvcG9ydC93aW4zMi5oLHYNCnJldHJpZXZpbmcgcmV2 -aXNpb24gMS40Ng0KZGlmZiAtYyAtcjEuNDYgd2luMzIuaA0KKioqIHNyYy9p -bmNsdWRlL3BvcnQvd2luMzIuaAkxNiBKdW4gMjAwNSAxNzo1Mzo1NCAtMDAw -MAkxLjQ2DQotLS0gc3JjL2luY2x1ZGUvcG9ydC93aW4zMi5oCTMwIEp1bCAy -MDA1IDE3OjM5OjM2IC0wMDAwDQoqKioqKioqKioqKioqKioNCioqKiAyNTYs -MjU4ICoqKioNCi0tLSAyNTYsMjYxIC0tLS0NCiAgDQogIC8qIGluIGJhY2tl -bmQvcG9ydC93aW4zMi9lcnJvci5jICovDQogIGV4dGVybiB2b2lkIF9kb3Nt -YXBlcnIodW5zaWduZWQgbG9uZyk7DQorIA0KKyAvKiBpbiBiYWNrZW5kL3Bv -cnQvd2luMzIvbG9jYWxlbWFwLmMgKi8NCisgZXh0ZXJuIGNoYXIgKnBnd2lu -MzJfbG9jYWxlbWFwKGNoYXIgKndpbmxvY2FsZSk7DQo= - -------_=_NextPart_001_01C5993C.1E1CB100 -Content-Type: application/octet-stream; - name="localemap.pl" -Content-Transfer-Encoding: base64 -Content-Description: localemap.pl -Content-Disposition: attachment; - filename="localemap.pl" - -IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj -IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMNCiMgbG9jYWxlbWFwLnBs -IC0tIEJ1aWxkIGxvY2FsZSBtYXAgYmFzZWQgb24gc291cmNlIGRhdGEgZmls -ZXMNCiMNCiMgQ29weXJpZ2h0IChjKSAyMDA1LCBQb3N0Z3JlU1FMIEdsb2Jh -bCBEZXZlbG9wbWVudCBHcm91cA0KIw0KIyAkUG9zdGdyZVNRTCQNCiMjIyMj -IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj -IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjDQojDQojIFRoZSBmaWxlIGlzbzYz -OSBjb250YWlucyBJU08tNjM5IGxhbmd1YWdlIGNvZGVzLCBjdXQtYW5kLXBh -c3RlIGZyb20NCiMgaHR0cDovL3d3dy53My5vcmcvV0FJL0VSL0lHL2VydC9p -c282MzkuaHRtLg0KIw0KIyBUaGUgZmlsZSBpc28zMTY2IGNvbnRhaW5zIElT -Ty0zMTY2IGNvdW50cnkgY29kZXMsIGN1dC1hbmQtcGFzdGUgZnJvbQ0KIyBo -dHRwOi8vd3d3Lm9hc2lzLW9wZW4ub3JnL2NvdmVyL2NvdW50cnkzMTY2Lmh0 -bWwsDQojDQoNCm9wZW4oSSwiaXNvNjM5IikgfHwgZGllICJDb3VsZCBub3Qg -b3BlbiBpc282MzlcbiI7DQp3aGlsZSAoPEk+KSB7DQoJY2hvbXA7DQoJZGll -ICJNYWxmb3JtYXR0ZWQgbGluZSBpbiBpc282MzkiIHVubGVzcyAvXihbQS1a -XXsyfSkgXCIoLispXCIkLzsNCgkkY29kZSA9IGxjICQxOw0KCSRsYW5nID0g -bGMgJDI7DQoJaWYgKCRsYW5nID1+IC9eKC4rKVwiIC8pIHsgJGxhbmcgPSAk -MTsgfQ0KCSRpc282Mzl7JGxhbmd9PSRjb2RlOw0KfQ0KY2xvc2UoSSk7DQoN -Cm9wZW4oSSwiaXNvMzE2NiIpIHx8IGRpZSAiQ291bGQgbm90IG9wZW4gaXNv -MzE2NlxuIjsNCndoaWxlICg8ST4pIHsNCgljaG9tcDsNCglkaWUgIk1hbGZv -cm1hdHRlZCBsaW5lIGluIGlzbzMxNjYiIHVubGVzcyAvXihbQS1aXXsyfSlc -cysoLispJC87DQoJJGNvZGUgPSBsYyAkMTsNCgkkY291bnRyeSA9IGxjICQy -Ow0KCSRpc28zMTY2eyRjb3VudHJ5fSA9ICRjb2RlOw0KfQ0KY2xvc2UoSSk7 -DQoNCnByaW50IDw8RU9GOw0KdHlwZWRlZiBzdHJ1Y3Qgew0KCWNoYXIgKmtl -eTsNCgljaGFyICp2YWw7DQp9IF9rZXl2YWw7DQoNCnN0YXRpYyBfa2V5dmFs -IGlzbzYzOVtdID0gew0KRU9GDQpmb3IgJGsgKHNvcnQga2V5cyAlaXNvNjM5 -KSB7DQoJcHJpbnQgIiB7XCIka1wiLFwiJGlzbzYzOXska31cIn0sXG4iOw0K -fQ0KcHJpbnQgIiB7TlVMTCxOVUxMfVxufTtcblxuIjsNCg0KcHJpbnQgInN0 -YXRpYyBfa2V5dmFsIGlzbzMxNjZbXSA9IHtcbiI7DQpmb3IgJGsgKHNvcnQg -a2V5cyAlaXNvMzE2Nikgew0KCXByaW50ICIge1wiJGtcIixcIiRpc28zMTY2 -eyRrfVwifSxcbiI7DQp9DQoNCnByaW50ICIge05VTEwsTlVMTH1cbn07XG5c -biI7DQo= - -------_=_NextPart_001_01C5993C.1E1CB100 -Content-Type: application/octet-stream; - name="localemap.c" -Content-Transfer-Encoding: base64 -Content-Description: localemap.c -Content-Disposition: attachment; - filename="localemap.c" - -LyoNCiAqIGxvY2FsZW1hcCAtIG1hcCB3aW4zMiBsb2NhbGUgbmFtZXMgdG8g -SUNVIHN0YW5kYXJkIG5hbWVzDQogKg0KICogQ29weXJpZ2h0IChDKSAyMDA1 -LCBQb3N0Z3JlU1FMIEdsb2JhbCBEZXZlbG9wbWVudCBHcm91cA0KICoNCiAq -ICRQb3N0Z3JlU1FMJA0KICovDQojaW5jbHVkZSAicG9zdGdyZXMuaCIgDQoN -CiNpbmNsdWRlICJsb2NhbGVtYXAuaCINCg0KDQpzdGF0aWMgY2hhciAqbWF0 -Y2hfbGFuZ3VhZ2UoY2hhciAqd2lubGFuZ3VhZ2UpDQp7DQoJX2tleXZhbCAq -a3Y7DQoNCglmb3IgKGt2ID0gaXNvNjM5OyBrdi0+a2V5OyBrdisrKSB7DQoJ -CWlmICghc3RyY2FzZWNtcChrdi0+a2V5LCB3aW5sYW5ndWFnZSkpDQoJCQly -ZXR1cm4ga3YtPnZhbDsNCgl9DQoJZXJlcG9ydChFUlJPUiwNCgkJCShlcnJt -c2coImNvdWxkIG5vdCBtYXRjaCBsYW5ndWFnZSBmb3IgJXMiLCB3aW5sYW5n -dWFnZSkpKTsNCglyZXR1cm4gTlVMTDsNCn0NCg0Kc3RhdGljIGNoYXIgKm1h -dGNoX2NvdW50cnkoY2hhciAqd2luY291bnRyeSkNCnsNCglfa2V5dmFsICpr -djsNCg0KCWZvciAoa3YgPSBpc28zMTY2OyBrdi0+a2V5OyBrdisrKSB7DQoJ -CWlmICghc3RyY2FzZWNtcChrdi0+a2V5LCB3aW5jb3VudHJ5KSkNCgkJCXJl -dHVybiBrdi0+dmFsOw0KCX0NCglyZXR1cm4gTlVMTDsNCn0NCg0KLyoNCiAq -IEF0dGVtcHQgdG8gY29udmVydCBhIHdpbjMyIGxvY2FsZSAoU3dlZGlzaF9T -d2VkZW4uMTI1MikNCiAqIHRvIElDVSBmb3JtYXQgKHN2X3NlKQ0KICovDQpz -dGF0aWMgY2hhciBvdXRidWZbMTZdOw0KY2hhciAqcGd3aW4zMl9sb2NhbGVt -YXAoY2hhciAqd2lubG9jYWxlKQ0Kew0KCWNoYXIgYnVmWzI1Nl07DQoJY2hh -ciAqdW5kZXJzY29yZTsNCgljaGFyICpkb3Q7DQoJY2hhciAqY291bnRyeTsN -Cg0KCWlmIChzdHJsZW4od2lubG9jYWxlKT4yNTUpIA0KCQllcmVwb3J0KEVS -Uk9SLA0KCQkJCShlcnJtc2dfaW50ZXJuYWwoImxvY2FsZSBuYW1lIHRvbyBs -b25nIikpKTsNCglzdHJjcHkoYnVmLHdpbmxvY2FsZSk7DQoNCgl1bmRlcnNj -b3JlID0gc3RyY2hyKGJ1ZiwnXycpOw0KCWlmICghdW5kZXJzY29yZSkNCgkJ -LyogT25seSBsYW5ndWFnZSBuYW1lICovDQoJCXJldHVybiBtYXRjaF9sYW5n -dWFnZShidWYpOw0KDQoJKnVuZGVyc2NvcmUgPSAwOw0KCXVuZGVyc2NvcmUr -KzsNCg0KCWRvdCA9IHN0cmNocih1bmRlcnNjb3JlLCcuJyk7DQoJaWYgKGRv -dCkNCgkJLyogSWYgY29kZXBhZ2UgaXMgaW5jbHVkZWQsIHdlIGp1c3QgaWdu -b3JlIGl0LiAqLw0KCQkqZG90ID0gMDsNCgkNCgljb3VudHJ5ID0gbWF0Y2hf -Y291bnRyeSh1bmRlcnNjb3JlKTsNCglpZiAoIWNvdW50cnkpDQoJCXJldHVy -biBtYXRjaF9sYW5ndWFnZShidWYpOw0KDQoJc3ByaW50ZihvdXRidWYsIiVz -XyVzIixtYXRjaF9sYW5ndWFnZShidWYpLGNvdW50cnkpOw0KCXJldHVybiBv -dXRidWY7DQp9DQo= - -------_=_NextPart_001_01C5993C.1E1CB100 -Content-Type: text/plain -Content-Disposition: inline -Content-Transfer-Encoding: 8bit -MIME-Version: 1.0 - - ----------------------------(end of broadcast)--------------------------- -TIP 4: Have you searched our list archives? - - http://archives.postgresql.org - -------_=_NextPart_001_01C5993C.1E1CB100-- - -From pgsql-hackers-owner+M72011@postgresql.org Wed Aug 24 15:37:17 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j7OJbH103190 - for ; Wed, 24 Aug 2005 15:37:17 -0400 (EDT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id D5152B21F6B; - Wed, 24 Aug 2005 19:37:12 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 34C03D7357 - for ; Wed, 24 Aug 2005 15:53:55 -0300 (ADT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 85477-03 - for ; - Wed, 24 Aug 2005 18:53:52 +0000 (GMT) -Received: from rodrick.geeknet.com.au (ns1.geeknet.com.au [220.244.63.182]) - by svr1.postgresql.org (Postfix) with ESMTP id 7C774D78F9 - for ; Wed, 24 Aug 2005 15:53:48 -0300 (ADT) -Subject: Re: [HACKERS] FreeBSD ICU was Win32 unicode vs ICU -MIME-Version: 1.0 -Date: Thu, 25 Aug 2005 04:53:47 +1000 -Content-Type: text/plain; - charset="us-ascii" -Message-ID: <5066E5A966339E42AA04BA10BA706AE50A938F@rodrick.geeknet.com.au> -content-class: urn:content-classes:message -X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0 -Thread-Topic: [HACKERS] FreeBSD ICU was Win32 unicode vs ICU -Thread-Index: AcWo0l6W575cyph2Twi7Fhfaa+w/tAACptMA -From: "John Hansen" -To: "Kevin McArthur" , - -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, hits=0.017 required=5 tests=[AWL=0.017] -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Transfer-Encoding: 8bit -X-MIME-Autoconverted: from quoted-printable to 8bit by candle.pha.pa.us id j7OJbH103190 -Status: OR - -Kevin McArthur Wrote: - -> Should the postgresql project also be looking at CLDR for -> cross-platform unicode support? - -Afaict, from the ICU website, ICU too uses CLDR. -Why reinvent the wheel? - -... John - ----------------------------(end of broadcast)--------------------------- -TIP 1: if posting/reading through Usenet, please send an appropriate - subscribe-nomail command to majordomo@postgresql.org so that your - message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M72581@postgresql.org Sat Sep 3 16:47:42 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j83Klg129281 - for ; Sat, 3 Sep 2005 16:47:42 -0400 (EDT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id CA484B21F00; - Sat, 3 Sep 2005 20:47:36 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 354A7D8B35 - for ; Sat, 3 Sep 2005 17:35:06 -0300 (ADT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 55971-07 - for ; - Sat, 3 Sep 2005 20:34:58 +0000 (GMT) -Received: from svana.org (svana.org [203.20.62.76]) - by svr1.postgresql.org (Postfix) with ESMTP id 4AFECD8AE7 - for ; Sat, 3 Sep 2005 17:34:56 -0300 (ADT) -Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian)) - id 1EBeim-0002xI-00; Sun, 04 Sep 2005 06:34:40 +1000 -Date: Sat, 3 Sep 2005 22:34:40 +0200 -From: Martijn van Oosterhout -To: Tom Lane -cc: Greg Stark , pgsql-hackers@postgresql.org -Subject: Locale implementation questions (was: [HACKERS] Proof of concept COLLATE support with patch) -Message-ID: <20050903203434.GA4281@svana.org> -Reply-To: Martijn van Oosterhout -References: <20050902130420.GA15466@svana.org> <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> -MIME-Version: 1.0 -Content-Type: multipart/signed; micalg=pgp-sha1; - protocol="application/pgp-signature"; boundary="mP3DRpeJDSE+ciuQ" -Content-Disposition: inline -In-Reply-To: <14696.1125675741@sss.pgh.pa.us> -User-Agent: Mutt/1.3.28i -X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6 -X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522 48A7 6394 F08A 0DC6 7BE6 -X-PGP-Key-URL: -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, hits=0.003 required=5 tests=[AWL=0.003] -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - ---mP3DRpeJDSE+ciuQ -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -Content-Transfer-Encoding: quoted-printable - -On Fri, Sep 02, 2005 at 11:42:21AM -0400, Tom Lane wrote: -> The objection is fundamentally that a platform-specific implementation -> cannot be our long-term goal, and so expending effort on creating one -> seems like a diversion. If there were a plan put forward showing how -> this is just a useful way-station, and we could see how we'd later get -> rid of the glibc dependency without throwing away the work already done, -> then it would be a different story. - -Well, my patch showed that useful locale work can be acheived with -precisely two functions: newlocale and strxfrm_l. - -I'm going to talk about two things: one, the code from Apple. Two, how -we present locale support to users. ---- -Now, it would be really nice to take Apple's implementation in Darwin -and use that. What I don't understand is the licence of the code in -Darwin. My interpretation is that stuff in: - -http://darwinsource.opendarwin.org/10.4.2/Libc-391/locale/ - -is Apple stuff under APSL, useless to us. And that stuff in: - -http://darwinsource.opendarwin.org/10.4.2/Libc-391/locale/FreeBSD/ - -are just patches to FreeBSD and this under the normal BSD license (no -big header claiming the licence change). The good news is that the -majority of what we need is in patch form. The bad news is that the hub -of the good stuff (newlocale, duplocale, freelocale) is under a big fat -APSL licence. - -Does anyone know if this code can be used at all by BSD projects or did -they blanket relicence everything? ---- -Now, I want to bring up some points relating to including a locale -library in PostgreSQL. Given that none of the BSDs seem really -interested in fixing the issue we'll have to do it ourselves (I don't -see anyone else doing it). We can save ourselves effort by basing it on -FreeBSDs locale code, because then we can use their datafiles, which we -*definitly* don't want to maintain ourselves. Now: - -1. FreeBSDs locale list is short, some 48 compared with glibc's 217. -Hopefully Apple can expand on that in a way we can use. But given the -difference we should probably give people a way of falling back to the -system libraries in case there's a locale we don't support. - -On the other hand, lots of locales are similar so maybe people can find -ones close enough to work. No, glibc and FreeBSD use different file -formats, so you can't copy them. - -Do we want this locale data just for collation, or do we want to be -able to use it for formatting monetary amounts too? This is even more -info to store. Lots of languages use ISO/IEC 14651 for order. - -2. Locale data needs to be combined with a charset and compiled to work -with the library. PostgreSQL supports at least 15 charsets but we don't -want to ship compiled versions of all of these (Debian learnt that the -hard way). So, how do we generate the files people need. - - a. Auto-compile on demand. First time a locale is referenced spawn -the compiler to create the locale, then continue. (Ugh) - b. Add a CREATE LOCALE english AS 'en_US' WITH CHARSET 'utf8'. Then -require the COLLATE clause to refer to this identifier. This has some -appeal, seperating the system names from the PostgreSQL names. It also -gives some info regarding charsets. - c. Should users be allowed to define new locales? - d. Should admins be required to create the external files using a -program, say pg_createlocale. - -Remember, if you use a latin1 locale to sort utf8 you'll get the wrong -result, so we want to avoid that. - -3. Compiled locale files are large. One UTF-8 locale datafile can -exceed a megabyte. Do we want the option of disabling it for small -systems? - -4. Do we want the option of running system locale in parallel with the -internal ones? - -5. I think we're going to have to deal with the very real possibility -that our locale database will not be as good as some of the system -provided ones. The question is how. This is quite unlike timezones -which are quite standardized and rarely change. That database is quite -well maintained. - -Would people object to a configure option that selected: - --with-locales=3Dinternal (use pg database) - --with-locales=3Dsystem (use system database for win32, glibc or Ma= -cOS X) - --with-locales=3Dnone (what we support now, which is neither) - -I don't think it will be much of an issue to support this, all the -functions take the same parameters and have almost the same names. - -6. Locales for SQL_ASCII. Seems to me you have two options, either -reject COLLATE altogether unless they specify a charset, or don't care -and let the user shoot themselves in the foot if they wish... - -BTW, this MacOS locale supports seems to be new for 10.4.2 according to -the CVS log info, can anyone confirm this? - -Anyway, I hope this post didn't bore too much. Locale support has been -one of those things that has bugged me for a long time and it would be -nice if there could be some real movement. - -Have a nice weekend, ---=20 -Martijn van Oosterhout http://svana.org/kleptog/ -> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a -> tool for doing 5% of the work and then sitting around waiting for someone -> else to do the other 95% so you can sue them. - ---mP3DRpeJDSE+ciuQ -Content-Type: application/pgp-signature -Content-Disposition: inline - ------BEGIN PGP SIGNATURE----- -Version: GnuPG v1.0.6 (GNU/Linux) -Comment: For info see http://www.gnupg.org - -iD8DBQFDGgjYIB7bNG8LQkwRAr6OAJ9uqNDKDQKWUAY4KiPAazHJ1TsVWwCeJ7sq -7hcILjdgZTQ4LjyPAhWnJwQ= -=5MT9 ------END PGP SIGNATURE----- - ---mP3DRpeJDSE+ciuQ-- - -From pgsql-hackers-owner+M72583@postgresql.org Sat Sep 3 17:54:58 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j83Lsv108133 - for ; Sat, 3 Sep 2005 17:54:57 -0400 (EDT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 3325AB21F00; - Sat, 3 Sep 2005 21:54:51 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id DA9AED8B14 - for ; Sat, 3 Sep 2005 18:44:52 -0300 (ADT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 88603-07 - for ; - Sat, 3 Sep 2005 21:44:51 +0000 (GMT) -Received: from stark.xeocode.com (stark.xeocode.com [216.58.44.227]) - by svr1.postgresql.org (Postfix) with ESMTP id 01864D8B50 - for ; Sat, 3 Sep 2005 18:44:50 -0300 (ADT) -Received: from localhost ([127.0.0.1] helo=stark.xeocode.com) - by stark.xeocode.com with smtp (Exim 3.36 #1 (Debian)) - id 1EBfog-0004U8-00; Sat, 03 Sep 2005 17:44:50 -0400 -To: Martijn van Oosterhout -cc: Tom Lane , Greg Stark , - pgsql-hackers@postgresql.org -Subject: Re: Locale implementation questions (was: [HACKERS] Proof of concept COLLATE support with patch) -References: <20050902130420.GA15466@svana.org> - <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> - <20050903203434.GA4281@svana.org> -In-Reply-To: <20050903203434.GA4281@svana.org> -From: Greg Stark -Organization: The Emacs Conspiracy; member since 1992 -Date: 03 Sep 2005 17:44:50 -0400 -Message-ID: <87oe79ygfh.fsf@stark.xeocode.com> -Lines: 39 -User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3 -MIME-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, hits=0.005 required=5 tests=[AWL=0.005] -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - -Martijn van Oosterhout writes: - -> 2. Locale data needs to be combined with a charset and compiled to work -> with the library. PostgreSQL supports at least 15 charsets but we don't -> want to ship compiled versions of all of these (Debian learnt that the -> hard way). So, how do we generate the files people need. - -That's just one of many lessons learned the hard way by distributions. Nor -will it be the last innovation in this area. - -I really find this instinct of wanting to reimplement large swaths of the OS -inside Postgres (and annoying detail-ridden swaths that are hard to get right -and continually evolving too) to be a bad idea. - -I can't believe it's harder to maintain an - -#ifdef HAVE_STRCOL_L -#else -#endif - -than it is to try to maintain an entire independent locale library. Nor is it -simpler for sysadmins to have to maintain an entirely separate set of locales -independently from the system locales. - -If you really are unhappy enough with OS setlocale implementations to want to -try to do this then it would be more helpful to do it outside of Postgres. -Package up the Apple setlocale library as a separate package that anyone can -install on Solaris, BSD, Linux or whatever. Then Postgres can just say "it -works fine with your OS library but your OS library might be very slow. Here's -a third-party library that you can install that is fast and may relieve any -problems you have with collation performance." - -But I think that's getting ahead of things. Until Postgres even supports -collations using the OS libraries you won't even know if that's even -necessary. - --- -greg - - ----------------------------(end of broadcast)--------------------------- -TIP 4: Have you searched our list archives? - - http://archives.postgresql.org - -From pgsql-hackers-owner+M72585@postgresql.org Sun Sep 4 08:44:01 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84Ci0127486 - for ; Sun, 4 Sep 2005 08:44:01 -0400 (EDT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 8ED1DB21F00; - Sun, 4 Sep 2005 12:43:55 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id D87D4D6E38 - for ; Sun, 4 Sep 2005 09:31:31 -0300 (ADT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 80362-07 - for ; - Sun, 4 Sep 2005 12:31:29 +0000 (GMT) -Received: from svana.org (svana.org [203.20.62.76]) - by svr1.postgresql.org (Postfix) with ESMTP id C0F8DD6D8D - for ; Sun, 4 Sep 2005 09:31:24 -0300 (ADT) -Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian)) - id 1EBteM-0005wf-00; Sun, 04 Sep 2005 22:31:06 +1000 -Date: Sun, 4 Sep 2005 14:31:05 +0200 -From: Martijn van Oosterhout -To: Greg Stark -cc: Tom Lane , pgsql-hackers@postgresql.org -Subject: Re: Locale implementation questions (was: [HACKERS] Proof of concept COLLATE support with patch) -Message-ID: <20050904123103.GA21198@svana.org> -Reply-To: Martijn van Oosterhout -References: <20050902130420.GA15466@svana.org> <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> <20050903203434.GA4281@svana.org> <87oe79ygfh.fsf@stark.xeocode.com> -MIME-Version: 1.0 -Content-Type: multipart/signed; micalg=pgp-sha1; - protocol="application/pgp-signature"; boundary="EeQfGwPcQSOJBaQU" -Content-Disposition: inline -In-Reply-To: <87oe79ygfh.fsf@stark.xeocode.com> -User-Agent: Mutt/1.3.28i -X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6 -X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522 48A7 6394 F08A 0DC6 7BE6 -X-PGP-Key-URL: -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, hits=0.003 required=5 tests=[AWL=0.003] -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - ---EeQfGwPcQSOJBaQU -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -Content-Transfer-Encoding: quoted-printable - -On Sat, Sep 03, 2005 at 05:44:50PM -0400, Greg Stark wrote: -> [...] Nor is it -> simpler for sysadmins to have to maintain an entirely separate set of loc= -ales -> independently from the system locales. - -Indeed, I was already coming up with mechanisms to determine what -locales the system uses and try to autogenerate them. I agree though, -it's not useful for systems that already have complete locale support. -Why add to the burden? - -Anyway, my reading of the specs says that we must support the syntax. -It doesn't say we need to support any orderings other than the default -(ie what we do now). - -> If you really are unhappy enough with OS setlocale implementations to wan= -t to -> try to do this then it would be more helpful to do it outside of Postgres. -> Package up the Apple setlocale library as a separate package that anyone = -can -> install on Solaris, BSD, Linux or whatever. Then Postgres can just say "it -> works fine with your OS library but your OS library might be very slow. H= -ere's -> a third-party library that you can install that is fast and may relieve a= -ny -> problems you have with collation performance." - -That's why I asked about the patches and files that Apple wrote. What -are the licence restrictions? Would we be able to download the, what, -20 files and distribute it as a library. Being APSL we couldn't include -it in the tarball, but it could be a pgfoundry project or something. - -If somebody knows a reason why this could not be done, speak up now because -my reading of the APSL licence tells me it's fine. - -> But I think that's getting ahead of things. Until Postgres even supports -> collations using the OS libraries you won't even know if that's even -> necessary. - -Well, I added COLLATE support for ORDER BY and CREATE INDEX and it -worked in under 200 lines. I'm thinking ahead and I don't think the -COLLATE rules are that hard. Implementing them seems a bit fiddly. It -may be easiest to consider COLLATE a non-associative operator. - -I'm still unsure if I should turn the string comparison operators into -three-argument functions. - -Anyway, I'll look into the library issue first. ---=20 -Martijn van Oosterhout http://svana.org/kleptog/ -> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a -> tool for doing 5% of the work and then sitting around waiting for someone -> else to do the other 95% so you can sue them. - ---EeQfGwPcQSOJBaQU -Content-Type: application/pgp-signature -Content-Disposition: inline - ------BEGIN PGP SIGNATURE----- -Version: GnuPG v1.0.6 (GNU/Linux) -Comment: For info see http://www.gnupg.org - -iD8DBQFDGukEIB7bNG8LQkwRAkeaAJ9rj5Tz7WPZpp+wWYWjTdjR68o8DwCfZzcZ -uB57noHmcnvefVoaw27bQ5Q= -=nLio ------END PGP SIGNATURE----- - ---EeQfGwPcQSOJBaQU-- - -From pgsql-hackers-owner+M72582@postgresql.org Sat Sep 3 17:46:57 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j83Lkr106932 - for ; Sat, 3 Sep 2005 17:46:57 -0400 (EDT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 6E192B21F00; - Sat, 3 Sep 2005 21:46:47 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 7CF09D72B7 - for ; Sat, 3 Sep 2005 18:36:14 -0300 (ADT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 77080-09 - for ; - Sat, 3 Sep 2005 21:36:13 +0000 (GMT) -Received: from stark.xeocode.com (stark.xeocode.com [216.58.44.227]) - by svr1.postgresql.org (Postfix) with ESMTP id 85FC0D6FEB - for ; Sat, 3 Sep 2005 18:36:09 -0300 (ADT) -Received: from localhost ([127.0.0.1] helo=stark.xeocode.com) - by stark.xeocode.com with smtp (Exim 3.36 #1 (Debian)) - id 1EBfgA-0004RS-00; Sat, 03 Sep 2005 17:36:02 -0400 -To: Tom Lane -cc: Greg Stark , Martijn van Oosterhout , - pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Proof of concept COLLATE support with patch -References: <20050902130420.GA15466@svana.org> - <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> -In-Reply-To: <14696.1125675741@sss.pgh.pa.us> -From: Greg Stark -Organization: The Emacs Conspiracy; member since 1992 -Date: 03 Sep 2005 17:36:02 -0400 -Message-ID: <87u0h1ygu5.fsf@stark.xeocode.com> -Lines: 36 -User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3 -MIME-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, hits=0.005 required=5 tests=[AWL=0.005] -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -Tom Lane writes: - -> Greg Stark writes: -> > I still doesn't get where the hostility towards this functionality comes from. -> -> We're not really willing to say "here is a piece of syntax REQUIRED -> BY THE SQL SPEC which we only support on some platforms". readline, -> O_DIRECT, and the like are a completely inappropriate analogy, because -> those are inherently platform-dependent (and not in the spec). - -But that's not the case at all. The syntax can be supported everywhere it -would just be somewhat faster on some platforms than others. It's already -reasonably fast on any platform that caches locale information which includes -glibc and presumably other free software libcs. It would be slightly faster if -there are _l functions. And much slower if the libc setlocale implementation -is braindead. But there's nothing wrong with saying "it's slow because your -libc is slow. Compile with this freely available library which has a better -implementation". The programming syntax would still be exactly 100% the same. - -> The objection is fundamentally that a platform-specific implementation -> cannot be our long-term goal, and so expending effort on creating one -> seems like a diversion. If there were a plan put forward showing how -> this is just a useful way-station, and we could see how we'd later get -> rid of the glibc dependency without throwing away the work already done, -> then it would be a different story. - -It's not like the actual calls to setlocale are going to be much code. One day -presumably some variant of these _l functions will become entirely standard. -In which case you're talking about potentially "throwing away" 50 lines of -code. The bulk of the code is going to be parsing and implementing the actual -syntax and behaviour of the SQL spec. And in any case I wouldn't expect it to -ever get thrown away. There will be people compiling on RH9 or similar vintage -systems for a long time. - --- -greg - - ----------------------------(end of broadcast)--------------------------- -TIP 3: Have you checked our extensive FAQ? - - http://www.postgresql.org/docs/faq - -From pgsql-hackers-owner+M72589@postgresql.org Sun Sep 4 13:20:26 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84HKQ121839 - for ; Sun, 4 Sep 2005 13:20:26 -0400 (EDT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id E4A85B21FBB; - Sun, 4 Sep 2005 17:20:23 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id C4CA7D7AEA - for ; Sun, 4 Sep 2005 14:07:02 -0300 (ADT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 63550-04 - for ; - Sun, 4 Sep 2005 17:07:01 +0000 (GMT) -Received: from mail.gmx.net (mail.gmx.net [213.165.64.20]) - by svr1.postgresql.org (Postfix) with SMTP id 02A4FD7A6F - for ; Sun, 4 Sep 2005 14:06:59 -0300 (ADT) -Received: (qmail invoked by alias); 04 Sep 2005 17:06:59 -0000 -Received: from dsl-082-083-230-040.arcor-ip.net (EHLO colt.pezone.net) [82.83.230.40] - by mail.gmx.net (mp033) with SMTP; 04 Sep 2005 19:06:59 +0200 -X-Authenticated: #495269 -From: Peter Eisentraut -To: pgsql-hackers@postgresql.org, Martijn van Oosterhout -Subject: Re: [HACKERS] Proof of concept COLLATE support with patch -Date: Sun, 4 Sep 2005 19:06:57 +0200 -User-Agent: KMail/1.8.1 -References: <20050902130420.GA15466@svana.org> -In-Reply-To: <20050902130420.GA15466@svana.org> -MIME-Version: 1.0 -Content-Type: text/plain; - charset="iso-8859-1" -Content-Transfer-Encoding: 7bit -Content-Disposition: inline -Message-ID: <200509041906.57721.peter_e@gmx.net> -X-Y-GMX-Trusted: 0 -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, hits=0.018 required=5 tests=[AWL=0.018] -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -Martijn van Oosterhout wrote: -> This is just a proof of concept patch. I didn't send it to -patches -> because as Tom pointed out, there's no hope of it getting in due to -> platform dependant behaviour. - -I think it would be best if we defined an internal API for plugging in -various kinds of locale support. Then you can hook in this -"newlocale", the Windows variant, ICU, or plain-old POSIX locale -support for backward compatibility. You already identified most of the -API functions. - --- -Peter Eisentraut -http://developer.postgresql.org/~petere/ - ----------------------------(end of broadcast)--------------------------- -TIP 9: In versions below 8.0, the planner will ignore your desire to - choose an index scan if your joining column's datatypes do not - match - -From pgsql-hackers-owner+M72592@postgresql.org Sun Sep 4 18:15:47 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84MFk103432 - for ; Sun, 4 Sep 2005 18:15:46 -0400 (EDT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id E623DB21FBF; - Sun, 4 Sep 2005 22:15:40 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id D671BD83B4 - for ; Sun, 4 Sep 2005 19:06:26 -0300 (ADT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 56030-04 - for ; - Sun, 4 Sep 2005 22:06:23 +0000 (GMT) -Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) - by svr1.postgresql.org (Postfix) with ESMTP id 44141D6F63 - for ; Sun, 4 Sep 2005 19:06:21 -0300 (ADT) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id j84M6Boq008947; - Sun, 4 Sep 2005 18:06:12 -0400 (EDT) -To: Greg Stark -cc: Martijn van Oosterhout , pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Proof of concept COLLATE support with patch -In-Reply-To: <87u0h1ygu5.fsf@stark.xeocode.com> -References: <20050902130420.GA15466@svana.org> <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> <87u0h1ygu5.fsf@stark.xeocode.com> -Comments: In-reply-to Greg Stark - message dated "03 Sep 2005 17:36:02 -0400" -Date: Sun, 04 Sep 2005 18:06:11 -0400 -Message-ID: <8946.1125871571@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, hits=0.006 required=5 tests=[AWL=0.006] -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -Greg Stark writes: -> But there's nothing wrong with saying "it's slow because your -> libc is slow. Compile with this freely available library which has a better -> implementation". - -The hole in that argument is the assumption that there *is* a freely -available library that can be used (where freely == BSD license). -We wouldn't be having this discussion if we knew of one. - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 1: if posting/reading through Usenet, please send an appropriate - subscribe-nomail command to majordomo@postgresql.org so that your - message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M72596@postgresql.org Sun Sep 4 19:27:26 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84NRQ109968 - for ; Sun, 4 Sep 2005 19:27:26 -0400 (EDT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id A427211ED4E0; - Sun, 4 Sep 2005 23:27:20 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 64BFED8B2F - for ; Sun, 4 Sep 2005 20:15:22 -0300 (ADT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 08469-05 - for ; - Sun, 4 Sep 2005 23:15:21 +0000 (GMT) -Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) - by svr1.postgresql.org (Postfix) with ESMTP id CE6D9D8AD4 - for ; Sun, 4 Sep 2005 20:15:19 -0300 (ADT) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id j84NFLVH009399; - Sun, 4 Sep 2005 19:15:21 -0400 (EDT) -To: Peter Eisentraut -cc: pgsql-hackers@postgresql.org, Martijn van Oosterhout -Subject: Re: [HACKERS] Proof of concept COLLATE support with patch -In-Reply-To: <200509041906.57721.peter_e@gmx.net> -References: <20050902130420.GA15466@svana.org> <200509041906.57721.peter_e@gmx.net> -Comments: In-reply-to Peter Eisentraut - message dated "Sun, 04 Sep 2005 19:06:57 +0200" -Date: Sun, 04 Sep 2005 19:15:21 -0400 -Message-ID: <9398.1125875721@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, hits=0.006 required=5 tests=[AWL=0.006] -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -Peter Eisentraut writes: -> I think it would be best if we defined an internal API for plugging in -> various kinds of locale support. - -Agreed ... - -> Then you can hook in this -> "newlocale", the Windows variant, ICU, or plain-old POSIX locale -> support for backward compatibility. - -If plain old POSIX actually did what we needed, we likely wouldn't be -having this discussion at all. POSIX doesn't give us enough visibility -of the locale's properties (in particular, which character set encoding -it wants). The performance penalties it imposes are pretty bad also, -though arguably secondary. - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 1: if posting/reading through Usenet, please send an appropriate - subscribe-nomail command to majordomo@postgresql.org so that your - message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M72599@postgresql.org Sun Sep 4 20:01:47 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j8501k113218 - for ; Sun, 4 Sep 2005 20:01:46 -0400 (EDT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 5FCFD11ED4E1; - Mon, 5 Sep 2005 00:01:40 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 3CE03D8A88 - for ; Sun, 4 Sep 2005 20:53:09 -0300 (ADT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 14738-01 - for ; - Sun, 4 Sep 2005 23:53:08 +0000 (GMT) -Received: from gateway.pjmodos.bounceme.net (gprsplusa185.isp.t-mobile.cz [62.141.20.185]) - by svr1.postgresql.org (Postfix) with ESMTP id B5606D8A4E - for ; Sun, 4 Sep 2005 20:53:03 -0300 (ADT) -Received: from modos ([10.12.0.96] ident=PJMODOS) - by gateway.pjmodos.bounceme.net with esmtp (Exim 3.35 #1 (Debian)) - id 1EC4IE-0004Le-00; Mon, 05 Sep 2005 01:52:58 +0200 -Message-ID: <431B88CD.8050904@seznam.cz> -Date: Mon, 05 Sep 2005 01:52:45 +0200 -From: Petr Jelinek -User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206) -X-Accept-Language: cs, en, en-us -MIME-Version: 1.0 -To: Tom Lane -cc: PostgreSQL-development -Subject: Re: [HACKERS] Proof of concept COLLATE support with patch -References: <20050902130420.GA15466@svana.org> <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> <87u0h1ygu5.fsf@stark.xeocode.com> <8946.1125871571@sss.pgh.pa.us> -In-Reply-To: <8946.1125871571@sss.pgh.pa.us> -Content-Type: text/plain; charset=windows-1250; format=flowed -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, hits=1.173 required=5 tests=[AWL=-0.253, - DNS_FROM_RFC_POST=1.376, FORGED_RCVD_HELO=0.05] -X-Spam-Level: * -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -Tom Lane wrote: -> -> The hole in that argument is the assumption that there *is* a freely -> available library that can be used (where freely == BSD license). -> We wouldn't be having this discussion if we knew of one. - -I see this discussion as another reason to use ICU, I mean complete -rewrite of locale handling to use ICU on all platforms. I know it's big -project but it's doable for 8.2 and it would virtually solve all locale -problems and could be base for new unicode/locale features. I am not -sure if this is the way postgres wants to go tho (having dependency on -such a big and uncommon library). - --- -Regards -Petr Jelinek (PJMODOS) - - ----------------------------(end of broadcast)--------------------------- -TIP 6: explain analyze is your friend - -From pgsql-hackers-owner+M72602@postgresql.org Mon Sep 5 04:02:37 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j8582Z126563 - for ; Mon, 5 Sep 2005 04:02:36 -0400 (EDT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 6640811EB014; - Mon, 5 Sep 2005 08:02:28 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 1FC36D7841 - for ; Mon, 5 Sep 2005 04:52:11 -0300 (ADT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 09506-02 - for ; - Mon, 5 Sep 2005 07:52:06 +0000 (GMT) -Received: from svana.org (svana.org [203.20.62.76]) - by svr1.postgresql.org (Postfix) with ESMTP id 0DC42D77AE - for ; Mon, 5 Sep 2005 04:52:06 -0300 (ADT) -Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian)) - id 1ECBlg-0001TS-00; Mon, 05 Sep 2005 17:51:52 +1000 -Date: Mon, 5 Sep 2005 09:51:52 +0200 -From: Martijn van Oosterhout -To: Tom Lane -cc: Greg Stark , pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Proof of concept COLLATE support with patch -Message-ID: <20050905075152.GA5278@svana.org> -Reply-To: Martijn van Oosterhout -References: <20050902130420.GA15466@svana.org> <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> <87u0h1ygu5.fsf@stark.xeocode.com> <8946.1125871571@sss.pgh.pa.us> -MIME-Version: 1.0 -Content-Type: multipart/signed; micalg=pgp-sha1; - protocol="application/pgp-signature"; boundary="MGYHOYXEY6WxJCY8" -Content-Disposition: inline -In-Reply-To: <8946.1125871571@sss.pgh.pa.us> -User-Agent: Mutt/1.3.28i -X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6 -X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522 48A7 6394 F08A 0DC6 7BE6 -X-PGP-Key-URL: -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, hits=0.003 required=5 tests=[AWL=0.003] -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - ---MGYHOYXEY6WxJCY8 -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -Content-Transfer-Encoding: quoted-printable - -On Sun, Sep 04, 2005 at 06:06:11PM -0400, Tom Lane wrote: -> Greg Stark writes: -> > But there's nothing wrong with saying "it's slow because your -> > libc is slow. Compile with this freely available library which has a be= -tter -> > implementation". ->=20 -> The hole in that argument is the assumption that there *is* a freely -> available library that can be used (where freely =3D=3D BSD license). -> We wouldn't be having this discussion if we knew of one. - -1. Use the one in Darwin just for the *BSDs and Solaris at least. It's -not great but it would probably work. - -2. Long term, transition to ICU (http://icu.sourceforge.net/) which is -the cross-platform internationalisation library used by Java. Looks -like Mono and Gnome/GTK are going to use this (or at least allow use -of) soon also. It uses the X licence AFAICS. It's a big pill right now -but it a year it could be installed standard on most linux systems. -It's at least avaiable everywhere now. - -Note, it's not compatable with POSIX locales so if we go there it'll be -an all or nothing switch. But if we intend to go there eventually, it -makes fiddling on our own library a waste of time. - -Incidently, I played with the code in Darwin and getting it to compile -on a system that already has extended locale support is, uh, tricky to -say the least. Lots of conflicting definitions. ---=20 -Martijn van Oosterhout http://svana.org/kleptog/ -> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a -> tool for doing 5% of the work and then sitting around waiting for someone -> else to do the other 95% so you can sue them. - ---MGYHOYXEY6WxJCY8 -Content-Type: application/pgp-signature -Content-Disposition: inline - ------BEGIN PGP SIGNATURE----- -Version: GnuPG v1.0.6 (GNU/Linux) -Comment: For info see http://www.gnupg.org - -iD8DBQFDG/kXIB7bNG8LQkwRAijSAJ4xv5JiigSwEe4jgBmxvqHN+GuOgQCfSEHp -C5tpavd58A8VEnf/BBUgfzo= -=gsNf ------END PGP SIGNATURE----- - ---MGYHOYXEY6WxJCY8-- - -From pgsql-hackers-owner+M72586@postgresql.org Sun Sep 4 09:37:44 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84Dbh100551 - for ; Sun, 4 Sep 2005 09:37:43 -0400 (EDT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 824CCB21F00; - Sun, 4 Sep 2005 13:37:37 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 413CCD7015 - for ; Sun, 4 Sep 2005 10:26:10 -0300 (ADT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 20529-10 - for ; - Sun, 4 Sep 2005 13:26:07 +0000 (GMT) -Received: from sraigw.sra.co.jp (sraigw.sra.co.jp [202.32.10.2]) - by svr1.postgresql.org (Postfix) with ESMTP id 12EF2D6F75 - for ; Sun, 4 Sep 2005 10:26:06 -0300 (ADT) -Received: from srascb.sra.co.jp (srascb [133.137.8.65]) - by sraigw.sra.co.jp (Postfix) with ESMTP - id 08EB262112; Sun, 4 Sep 2005 22:26:09 +0900 (JST) -Received: from srascb.sra.co.jp (localhost [127.0.0.1]) - by localhost.sra.co.jp (Postfix) with ESMTP - id D1D9610CD06; Sun, 4 Sep 2005 22:26:09 +0900 (JST) -Received: from sranhm.sra.co.jp (sranhm [133.137.44.16]) - by srascb.sra.co.jp (Postfix) with ESMTP - id ABF1210CD04; Sun, 4 Sep 2005 22:26:09 +0900 (JST) -Received: from localhost (sraihb-hub.sra.co.jp [133.137.8.6]) - by sranhm.sra.co.jp (8.9.3+3.2W/3.7W-srambox) with ESMTP id WAA12274; - Sun, 4 Sep 2005 22:26:09 +0900 -Date: Sun, 04 Sep 2005 22:25:36 +0900 (JST) -Message-ID: <20050904.222536.39155679.ishii@sraoss.co.jp> -To: kleptog@svana.org -cc: tgl@sss.pgh.pa.us, gsstark@mit.edu, pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Locale implementation questions -From: Tatsuo Ishii -In-Reply-To: <20050903203434.GA4281@svana.org> -References: <87k6hzzems.fsf@stark.xeocode.com> - <14696.1125675741@sss.pgh.pa.us> - <20050903203434.GA4281@svana.org> -X-Mailer: Mew version 2.3 on Emacs 20.7 / Mule 4.1 - =?iso-2022-jp?B?KBskQjAqGyhCKQ==?= -MIME-Version: 1.0 -Content-Type: Text/Plain; charset=us-ascii -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, hits=0.028 required=5 tests=[AWL=0.028] -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -> 3. Compiled locale files are large. One UTF-8 locale datafile can -> exceed a megabyte. Do we want the option of disabling it for small -> systems? - -To avoid the problem, you could dynmically load the compiled -tables. The charset conversion tables are handled similar way. - -Also I think it's important to allow user defined collate data. To -implement the CREATE COLLATE syntax, we need to have that capability -anyway. - -> 4. Do we want the option of running system locale in parallel with the -> internal ones? -> -> 5. I think we're going to have to deal with the very real possibility -> that our locale database will not be as good as some of the system -> provided ones. The question is how. This is quite unlike timezones -> which are quite standardized and rarely change. That database is quite -> well maintained. -> -> Would people object to a configure option that selected: -> --with-locales=internal (use pg database) -> --with-locales=system (use system database for win32, glibc or MacOS X) -> --with-locales=none (what we support now, which is neither) -> -> I don't think it will be much of an issue to support this, all the -> functions take the same parameters and have almost the same names. - -To be honest, I don't understand why we have to rely on (often broken) -system locales. I don't think building our own locale data is too -hard, and once we make up it, the maintenace cost will be very small -since it should not be changed regularly. Moreover we could enjoy the -benefit that PostgreSQL handles collations in a corret manner on any -platform which PostgreSQL supports. - -> 6. Locales for SQL_ASCII. Seems to me you have two options, either -> reject COLLATE altogether unless they specify a charset, or don't care -> and let the user shoot themselves in the foot if they wish... -> -> BTW, this MacOS locale supports seems to be new for 10.4.2 according to -> the CVS log info, can anyone confirm this? -> -> Anyway, I hope this post didn't bore too much. Locale support has been -> one of those things that has bugged me for a long time and it would be -> nice if there could be some real movement. - -Right. We Japanese (and probably Chinese too) have been bugged by the -broken mutibyte locales for long time. Using C locale help us to a -certain extent, but for Unicode we need correct locale data, othewise -the sorted data will be completely chaos. --- -SRA OSS, Inc. Japan -Tatsuo Ishii - ----------------------------(end of broadcast)--------------------------- -TIP 1: if posting/reading through Usenet, please send an appropriate - subscribe-nomail command to majordomo@postgresql.org so that your - message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M72587@postgresql.org Sun Sep 4 11:13:07 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84FD5107053 - for ; Sun, 4 Sep 2005 11:13:06 -0400 (EDT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id D1EC5B21F00; - Sun, 4 Sep 2005 15:13:04 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 8823DD723A - for ; Sun, 4 Sep 2005 12:01:47 -0300 (ADT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 65135-01 - for ; - Sun, 4 Sep 2005 15:01:43 +0000 (GMT) -Received: from svana.org (svana.org [203.20.62.76]) - by svr1.postgresql.org (Postfix) with ESMTP id 5896DD70F7 - for ; Sun, 4 Sep 2005 12:01:40 -0300 (ADT) -Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian)) - id 1EBvzd-0006eP-00; Mon, 05 Sep 2005 01:01:13 +1000 -Date: Sun, 4 Sep 2005 17:01:13 +0200 -From: Martijn van Oosterhout -To: Tatsuo Ishii -cc: tgl@sss.pgh.pa.us, gsstark@mit.edu, pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Locale implementation questions -Message-ID: <20050904150055.GB21198@svana.org> -Reply-To: Martijn van Oosterhout -References: <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> <20050903203434.GA4281@svana.org> <20050904.222536.39155679.ishii@sraoss.co.jp> -MIME-Version: 1.0 -Content-Type: multipart/signed; micalg=pgp-sha1; - protocol="application/pgp-signature"; boundary="ZfOjI3PrQbgiZnxM" -Content-Disposition: inline -In-Reply-To: <20050904.222536.39155679.ishii@sraoss.co.jp> -User-Agent: Mutt/1.3.28i -X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6 -X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522 48A7 6394 F08A 0DC6 7BE6 -X-PGP-Key-URL: -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, hits=0.003 required=5 tests=[AWL=0.003] -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - ---ZfOjI3PrQbgiZnxM -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -Content-Transfer-Encoding: quoted-printable - -On Sun, Sep 04, 2005 at 10:25:36PM +0900, Tatsuo Ishii wrote: -> > 3. Compiled locale files are large. One UTF-8 locale datafile can -> > exceed a megabyte. Do we want the option of disabling it for small -> > systems? ->=20 -> To avoid the problem, you could dynmically load the compiled -> tables. The charset conversion tables are handled similar way. - -That's not the point, ofcourse they are loaded dynamically. The -question is, when do we create the files in the first place. There are -48*15 =3D 750 combinations which would amount to tens of megabytes of -essentially useless data. *When* you create the files is an important -question. Compile time is out. - -Charset conversion is completely different, there just arn't that many -combinations. - -> Also I think it's important to allow user defined collate data. To -> implement the CREATE COLLATE syntax, we need to have that capability -> anyway. - -Most OS's allow you to create collate data yourself anyway, why do we -need to implement this too? - -> To be honest, I don't understand why we have to rely on (often broken) -> system locales. I don't think building our own locale data is too -> hard, and once we make up it, the maintenace cost will be very small -> since it should not be changed regularly. Moreover we could enjoy the -> benefit that PostgreSQL handles collations in a corret manner on any -> platform which PostgreSQL supports. - -You say building our own locale data is not hard. I disagree, it's a -waste of time we can do without. Unless you know the language yourself -you cannot check changes made by anybody else. If there's an error in -locale ordering, take it up with your OS distributor. - -I also think we open ourselves to questions like: - -1. My locale is supported by the system but not by PostgreSQL, why? -2. My locale was supported last release but not this one, why? -3. Why does PostgreSQL sort differently from 'sort' or any other app on -my system? - -> Right. We Japanese (and probably Chinese too) have been bugged by the -> broken mutibyte locales for long time. Using C locale help us to a -> certain extent, but for Unicode we need correct locale data, othewise -> the sorted data will be completely chaos. - -Ok, is glibc still wrong or are they just implementing the unicode -standard and that's what's wrong. - -All I'm saying is that we need to allow use of system locales until our -native locale support is mature. In the end something like ICU -(http://icu.sourceforge.net/) will end up obsoleting us. Nobody (in -free-software anyway) uses it yet, but eventually it may be viable to -require that to allow system independant locales. ---=20 -Martijn van Oosterhout http://svana.org/kleptog/ -> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a -> tool for doing 5% of the work and then sitting around waiting for someone -> else to do the other 95% so you can sue them. - ---ZfOjI3PrQbgiZnxM -Content-Type: application/pgp-signature -Content-Disposition: inline - ------BEGIN PGP SIGNATURE----- -Version: GnuPG v1.0.6 (GNU/Linux) -Comment: For info see http://www.gnupg.org - -iD8DBQFDGwwmIB7bNG8LQkwRAmKxAJ9MccMVTLVLqW7c9Lsa8FzqEMvNGgCdGFgg -nsvl/ZIAK/Qe1XpkMOItRFM= -=h0V5 ------END PGP SIGNATURE----- - ---ZfOjI3PrQbgiZnxM-- - -From pgsql-hackers-owner+M72590@postgresql.org Sun Sep 4 15:15:31 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84JFU107171 - for ; Sun, 4 Sep 2005 15:15:31 -0400 (EDT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 950F5B21FBE; - Sun, 4 Sep 2005 19:15:26 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id AA4C5D85BF - for ; Sun, 4 Sep 2005 16:02:50 -0300 (ADT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 51227-06 - for ; - Sun, 4 Sep 2005 19:02:48 +0000 (GMT) -Received: from stark.xeocode.com (stark.xeocode.com [216.58.44.227]) - by svr1.postgresql.org (Postfix) with ESMTP id CC650D8580 - for ; Sun, 4 Sep 2005 16:02:46 -0300 (ADT) -Received: from localhost ([127.0.0.1] helo=stark.xeocode.com) - by stark.xeocode.com with smtp (Exim 3.36 #1 (Debian)) - id 1EBzlN-0003LY-00; Sun, 04 Sep 2005 15:02:45 -0400 -To: Tatsuo Ishii -cc: kleptog@svana.org, tgl@sss.pgh.pa.us, gsstark@mit.edu, - pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Locale implementation questions -References: <87k6hzzems.fsf@stark.xeocode.com> - <14696.1125675741@sss.pgh.pa.us> <20050903203434.GA4281@svana.org> - <20050904.222536.39155679.ishii@sraoss.co.jp> -In-Reply-To: <20050904.222536.39155679.ishii@sraoss.co.jp> -From: Greg Stark -Organization: The Emacs Conspiracy; member since 1992 -Date: 04 Sep 2005 15:02:45 -0400 -Message-ID: <87fysky7u2.fsf@stark.xeocode.com> -Lines: 22 -User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3 -MIME-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, hits=0.005 required=5 tests=[AWL=0.005] -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - -Tatsuo Ishii writes: - -> To be honest, I don't understand why we have to rely on (often broken) -> system locales. I don't think building our own locale data is too -> hard, and once we make up it, the maintenace cost will be very small -> since it should not be changed regularly. Moreover we could enjoy the -> benefit that PostgreSQL handles collations in a corret manner on any -> platform which PostgreSQL supports. - -I think it's sheer madness to try to reproduce large swaths of the OS inside -Postgres because you're unhappy with the quality of the OS implementation. You -should be asking yourself why OS vendors have such a hard time getting this -stuff right and why would Postgres do any better. Wouldn't that work be better -spent improving the database functionality of Postgres? - -Or at least better spent improving the locale support for the entire OS? It -would be positively awful if every application on my system had its own locale -database each of which had its own set of bugs and its own feature set. - --- -greg - - ----------------------------(end of broadcast)--------------------------- -TIP 3: Have you checked our extensive FAQ? - - http://www.postgresql.org/docs/faq - -From pgsql-hackers-owner+M72597@postgresql.org Sun Sep 4 19:28:34 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84NSY110074 - for ; Sun, 4 Sep 2005 19:28:34 -0400 (EDT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 7898611ED4E0; - Sun, 4 Sep 2005 23:28:28 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id BE640D8910 - for ; Sun, 4 Sep 2005 20:19:37 -0300 (ADT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 09430-04 - for ; - Sun, 4 Sep 2005 23:19:37 +0000 (GMT) -Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) - by svr1.postgresql.org (Postfix) with ESMTP id 7DE22D8864 - for ; Sun, 4 Sep 2005 20:19:34 -0300 (ADT) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id j84NJclc009441; - Sun, 4 Sep 2005 19:19:38 -0400 (EDT) -To: Greg Stark -cc: Tatsuo Ishii , kleptog@svana.org, - pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Locale implementation questions -In-Reply-To: <87fysky7u2.fsf@stark.xeocode.com> -References: <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> <20050903203434.GA4281@svana.org> <20050904.222536.39155679.ishii@sraoss.co.jp> <87fysky7u2.fsf@stark.xeocode.com> -Comments: In-reply-to Greg Stark - message dated "04 Sep 2005 15:02:45 -0400" -Date: Sun, 04 Sep 2005 19:19:38 -0400 -Message-ID: <9440.1125875978@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, hits=0.006 required=5 tests=[AWL=0.006] -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -Greg Stark writes: -> I think it's sheer madness to try to reproduce large swaths of the OS -> inside Postgres because you're unhappy with the quality of the OS -> implementation. You should be asking yourself why OS vendors have such -> a hard time getting this stuff right - -In the case of the *BSDs, it's pretty obviously because they don't care. - -> and why would Postgres do any better - -In the first place, we do care, and in the second place, having to deal -with only one set of locale bugs would in itself be a huge advance over -where we are now. - -We went over to maintaining our own timezone code for more or less the -same reasons, and in hindsight that was obviously the right decision. -Locale support is a bigger chunk, no doubt about it, but we also have -a lot of motivation. - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 2: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M72619@postgresql.org Mon Sep 5 18:14:28 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j85MER115087 - for ; Mon, 5 Sep 2005 18:14:27 -0400 (EDT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 68D9F11EB015; - Mon, 5 Sep 2005 22:14:19 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 92268D6FB8 - for ; Mon, 5 Sep 2005 18:59:42 -0300 (ADT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 29325-07 - for ; - Mon, 5 Sep 2005 21:59:38 +0000 (GMT) -Received: from svana.org (svana.org [203.20.62.76]) - by svr1.postgresql.org (Postfix) with ESMTP id 98C2CD7A97 - for ; Mon, 5 Sep 2005 18:59:33 -0300 (ADT) -Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian)) - id 1ECP02-0004SQ-00; Tue, 06 Sep 2005 07:59:34 +1000 -Date: Mon, 5 Sep 2005 23:59:33 +0200 -From: Martijn van Oosterhout -To: pgsql-hackers@postgresql.org -Subject: [HACKERS] Install Darwin's locale library on your system :) -Message-ID: <20050905215928.GB5278@svana.org> -Reply-To: Martijn van Oosterhout -MIME-Version: 1.0 -Content-Type: multipart/signed; micalg=pgp-sha1; - protocol="application/pgp-signature"; boundary="hHWLQfXTYDoKhP50" -Content-Disposition: inline -User-Agent: Mutt/1.3.28i -X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6 -X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522 48A7 6394 F08A 0DC6 7BE6 -X-PGP-Key-URL: -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, hits=0.003 required=5 tests=[AWL=0.003] -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - ---hHWLQfXTYDoKhP50 -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -Content-Transfer-Encoding: quoted-printable - -Well, it was pointed out the other day that the Darwin C library -supports the non-standard extensions to the POSIX locale interface and -that this might be ported to other systems so PostgreSQL could use it. - -So, I have written a few scripts which download the libc and locale -library from darwinsource, shuffle some files around and build the -result into a library called libdummylocale.so. It basically completely -replaces your locale support on whatever system you use it on. - -It's all under the APSL, though some parts may be BSD licenced. - -Let me say right now, the locale support here sucks, no two ways about -it. It doesn't support a single UTF-8 locale. Oh, it lets you specify -them, but when you ask for the CHARSET it still says US-ASCII. It does -support a number of other different charsets. (Not for collation -though). - -So my challenge to those people who think maintaining a locale library -is easy: make *one* locale in FreeBSD (or Darwin or this lib) support -full UTF-8 collation in whatever locale and/or charset you choose. It's -all downhill from there. - -While it builds simple programs, I don't think it's totally safe. You'd -need to rename the headers at least. And building on Darwin will -probably blow up due to the way it plays fast and loose with Darwin -specific #defines. But it's a beginning if anyone is interested. It -builds in my glibc system. - -I'm going to drop the idea of making a locale library, there's just -nothing good enough. glibc is the only thing that comes close. From here -on I'm going to work on COLLATE for systems that support xlocale, with -an eye on ICU if/when it becomes standard enough. - -Download: http://svana.org/kleptog/pgsql/dummylocale.tar.gz - -Have a nice day, ---=20 -Martijn van Oosterhout http://svana.org/kleptog/ -> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a -> tool for doing 5% of the work and then sitting around waiting for someone -> else to do the other 95% so you can sue them. - ---hHWLQfXTYDoKhP50 -Content-Type: application/pgp-signature -Content-Disposition: inline - ------BEGIN PGP SIGNATURE----- -Version: GnuPG v1.0.6 (GNU/Linux) -Comment: For info see http://www.gnupg.org - -iD8DBQFDHL++IB7bNG8LQkwRAjvyAJ98Hwm6Ak7zSdEU7shrzxbvMJJp/QCfcJSq -ZseeCwJGWAs+3MEPEr8u2dM= -=rr4R ------END PGP SIGNATURE----- - ---hHWLQfXTYDoKhP50-- - -From pgsql-patches-owner+M17366@postgresql.org Wed Sep 7 12:16:02 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j87GG1104073 - for ; Wed, 7 Sep 2005 12:16:01 -0400 (EDT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id D3FD711EB015; - Wed, 7 Sep 2005 16:15:58 +0000 (GMT) -X-Original-To: pgsql-patches-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 326F8D7B55 - for ; Wed, 7 Sep 2005 13:11:41 -0300 (ADT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 66946-04 - for ; - Wed, 7 Sep 2005 16:11:25 +0000 (GMT) -Received: from svana.org (svana.org [203.20.62.76]) - by svr1.postgresql.org (Postfix) with ESMTP id D88A6D6E3B - for ; Wed, 7 Sep 2005 13:11:20 -0300 (ADT) -Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian)) - id 1ED2W5-00052g-00; Thu, 08 Sep 2005 02:11:17 +1000 -Date: Wed, 7 Sep 2005 18:11:17 +0200 -From: Martijn van Oosterhout -To: pgsql-patches@postgresql.org -Subject: [PATCHES] For review: Initial support for COLLATE -Message-ID: <20050907161112.GA10273@svana.org> -Reply-To: Martijn van Oosterhout -MIME-Version: 1.0 -Content-Type: multipart/signed; micalg=pgp-sha1; - protocol="application/pgp-signature"; boundary="ibTvN161/egqYuK8" -Content-Disposition: inline -User-Agent: Mutt/1.3.28i -X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6 -X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522 48A7 6394 F08A 0DC6 7BE6 -X-PGP-Key-URL: -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, hits=0.003 required=5 tests=[AWL=0.003] -X-Mailing-List: pgsql-patches -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-patches-owner@postgresql.org -Status: OR - - ---ibTvN161/egqYuK8 -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -Content-Transfer-Encoding: quoted-printable - -[Please CC any replies, thanks] - -This patch is the beginnings of support for COLLATE. I need to do some -other work for a few days so I'm posting here to get some initial -reviews. Various parts are marked [done] and [not done]. - -The steps involved are: - -- Add COLLATE to grammer as part of expression tree. New CollateClause - node stores the relevent facts. [done] - -- parse_expr goes through tree and determines the appropriate COLLATE - state for each node as per SQL spec. [done] - - Note: PostgreSQL doesn't really have a way to identify text-like - types and there's no real need to anyway. The implementation allows - any node to have a collate state. This can be used for other things, - see below. - -- CREATE COLLATE statement [not done, currently added on the fly] - -- Two new datatypes, pg_locale_t and pg_localedata_t. The former - represents a locale (and will eventually have an OID). The latter is - an anonymous cookie that stores locale specific information and is - passed to functions that need it. It may be that once done, this - latter type will vanish again. [done] - -- Several utility functions in the new file pg_xlocale.c for use by the - rest of the system. [done] - -- Add boolean column 'proislocalized' to pg_proc which indicates if the - output of this function is affected by the LOCALE. eg textcat doesn't - care but textle does. [done] This is for: - - a) So the parser can complain about functions that look at the - locale/collate order but it's not clear from the arguments (state - None per SQL spec) and it's not specified explicitly. [not done] - - b) So when a column or function is indexed, the index code knows if - the locale is relevent to sorting order. This is particularly - interesting for btree indexes on character strings. [not done] - - Currently I've marked 58 functions as being locale sensetive, but - that list will need careful going over. To some extent it can be - checked automatically by examining which backend functions use the - new PG_GETLOCALE() macro. - -- Check for correct encoding in loaded locales [not done] - -- Check the partial indexes do the right thing when matching - expressions. [not done] - -- Docs, regression, etc... - -- make check: right now I'm getting some regressions in the rules - and plpgsql, very odd... Possibly due to the fact that rules get - collate nodes with locales that don't persist across invokations. - -Goals: - -- This setup extends the SQL spec a bit, in the sense that COLLATE can - be attached to anything. It is my intention to allow functions such - as to_char() and to_timestamp() to be localized. eg: - -test=3D# select cash_out('1.00'::money collate 'nl_NL'), cash_out('1.00'::m= -oney collate 'en_AU'); - cash_out | cash_out=20 -----------+---------- - EUR1,00 | $1.00 -(1 row) - -- Should LOCALE be created as a synonym for COLLATE? It reads more - naturally. - -- Currently LC_COLLATE is fixed at initdb and LC_NUMERIC and - LC_CURRENCY can be altered. The idea is that eventually even - LC_COLLATE can be altered anytime (from the users point view anyway), - as any objects that care will store the collate order they want and - can execute functions as appropriate. - -- Eventually once the transition to full locale support is complete, - change the backend so it always runs under locale C and only - functions on userdata are affected by locales. This means that - unquoted identifiers when converted to lowercase will be lowered as - per ASCII rules. Judging by [1] this is what people want, but - feedback would be nice. - -[1] http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg39742.html - - Hence, right now, names compare using normal strcmp, but text, - varchar, etc use strcoll. - -- The patch as it currently stands won't compile at all without system - xlocale support. The goal is to provide some level of backward - compatability where the COLLATE clause can be used, but only to - affect functions like to_char(). Changing COLLATE order would be - forbidden (ie just like now). - -- Eventually, once the parts relevent to locales are sufficiently - abstracted, look into ICU to plug it in. Unfortunatly, it's a - completely different model (all utf-16) so that's another phase - altogether. - -- The default type output functions should never be locale specific. - This is to avoid issues with pgdump and frontends. Create a function - 'localize(anyelement)' to "do the obvious" to force it to happen. - -Download:=20 - Compressed 57K, uncompressed >500K but that's due to rewriting the - whole of pg_proc. The important code is not so big. Against todays - CVS. - -http://svana.org/kleptog/pgsql/collate2.patch.gz - -Examples: - -test=3D# SELECT text('a') < text('B') COLLATE 'C', text('a') < text('B') CO= -LLATE 'en_US.UTF-8'; - ?column? | ?column?=20 -----------+---------- - f | t -(1 row) - -test=3D# SELECT text('A') < text('b') COLLATE 'C', text('A') < text('b') CO= -LLATE 'en_US.UTF-8'; - ?column? | ?column?=20 -----------+---------- - t | t -(1 row) - -test=3D# SELECT text('A') COLLATE 'en_US.UTF-8' < text('b') COLLATE 'C'; -ERROR: Conflicting COLLATE clauses - -Have a nice day, ---=20 -Martijn van Oosterhout http://svana.org/kleptog/ -> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a -> tool for doing 5% of the work and then sitting around waiting for someone -> else to do the other 95% so you can sue them. - ---ibTvN161/egqYuK8 -Content-Type: application/pgp-signature -Content-Disposition: inline - ------BEGIN PGP SIGNATURE----- -Version: GnuPG v1.0.6 (GNU/Linux) -Comment: For info see http://www.gnupg.org - -iD8DBQFDHxEgIB7bNG8LQkwRAqTqAJ0TOH3RkxnYHJmCPhjjYha0idsEoACfTjY0 -EP4a7CtIaZ0Ar0dyuPja3h8= -=8zan ------END PGP SIGNATURE----- - ---ibTvN161/egqYuK8-- - -From pgsql-patches-owner+M17368@postgresql.org Wed Sep 7 16:31:39 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j87KVd101436 - for ; Wed, 7 Sep 2005 16:31:39 -0400 (EDT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 847AF11EB01E; - Wed, 7 Sep 2005 20:31:32 +0000 (GMT) -X-Original-To: pgsql-patches-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 22823D7D5C - for ; Wed, 7 Sep 2005 16:12:20 -0300 (ADT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 19251-09 - for ; - Wed, 7 Sep 2005 19:12:13 +0000 (GMT) -Received: from mail.gmx.net (mail.gmx.de [213.165.64.20]) - by svr1.postgresql.org (Postfix) with SMTP id 05BA7D7B7E - for ; Wed, 7 Sep 2005 16:12:13 -0300 (ADT) -Received: (qmail invoked by alias); 07 Sep 2005 19:12:14 -0000 -Received: from dsl-082-083-195-234.arcor-ip.net (EHLO colt.pezone.net) [82.83.195.234] - by mail.gmx.net (mp022) with SMTP; 07 Sep 2005 21:12:14 +0200 -X-Authenticated: #495269 -From: Peter Eisentraut -To: Martijn van Oosterhout -Subject: Re: [PATCHES] For review: Initial support for COLLATE -Date: Wed, 7 Sep 2005 21:12:12 +0200 -User-Agent: KMail/1.8.1 -References: <20050907161112.GA10273@svana.org> -In-Reply-To: <20050907161112.GA10273@svana.org> -cc: pgsql-patches@postgresql.org -MIME-Version: 1.0 -Content-Type: text/plain; - charset="iso-8859-1" -Content-Transfer-Encoding: 7bit -Content-Disposition: inline -Message-ID: <200509072112.12955.peter_e@gmx.net> -X-Y-GMX-Trusted: 0 -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, hits=0.044 required=5 tests=[AWL=-0.006, - FORGED_RCVD_HELO=0.05] -X-Mailing-List: pgsql-patches -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-patches-owner@postgresql.org -Status: OR - -Martijn van Oosterhout wrote: -> - Should LOCALE be created as a synonym for COLLATE? It reads more -> naturally. - -No, and in fact the terminology mixup in your patch and description -concerns me. If you are talking about collation, then the data types, -system catalog columns, etc. should talk about collation, not about -"locale", because that encompasses a number of other things that can be -handled independent of the collation order. - --- -Peter Eisentraut -http://developer.postgresql.org/~petere/ - ----------------------------(end of broadcast)--------------------------- -TIP 9: In versions below 8.0, the planner will ignore your desire to - choose an index scan if your joining column's datatypes do not - match - -From pgsql-hackers-owner+M77964=pgman=candle.pha.pa.us@postgresql.org Tue Dec 27 06:30:14 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jBRDUET16504 - for ; Tue, 27 Dec 2005 08:30:14 -0500 (EST) -Received: from postgresql.org (postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 3BCBA67A57F - for ; Tue, 27 Dec 2005 09:30:14 -0400 (AST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by postgresql.org (Postfix) with ESMTP id 1DB3B9DC859 - for ; Tue, 27 Dec 2005 09:29:47 -0400 (AST) -Received: from postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 89101-04 - for ; - Tue, 27 Dec 2005 09:29:50 -0400 (AST) -X-Greylist: from auto-whitelisted by SQLgrey- -Received: from svana.org (svana.org [203.20.62.76]) - by postgresql.org (Postfix) with ESMTP id 319839DC8A5 - for ; Tue, 27 Dec 2005 09:29:44 -0400 (AST) -Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian)) - id 1ErEte-0001lm-00; Wed, 28 Dec 2005 00:29:46 +1100 -Date: Tue, 27 Dec 2005 14:29:46 +0100 -From: Martijn van Oosterhout -To: pgsql-hackers@postgresql.org -Subject: [HACKERS] Proposed COLLATE implementation -Message-ID: <20051227132941.GA32404@svana.org> -Reply-To: Martijn van Oosterhout -MIME-Version: 1.0 -Content-Type: multipart/signed; micalg=pgp-sha1; - protocol="application/pgp-signature"; boundary="W/nzBZO5zC0uMSeA" -Content-Disposition: inline -User-Agent: Mutt/1.3.28i -X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6 -X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522 48A7 6394 F08A 0DC6 7BE6 -X-PGP-Key-URL: -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.08 required=5 tests=[AWL=0.080] -X-Spam-Score: 0.08 -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - ---W/nzBZO5zC0uMSeA -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -Content-Transfer-Encoding: quoted-printable - -Greetings all, - -If you're not interested in COLLATE, operator classes or related -things, stop now, this is quite a long email. - -Firstly, status. PostgreSQL doesn't really support collations at all. -The order of strings is defined at initdb time by the locale then and -cannot be changed later. We allow lists to be sorted in either -ascending or decending order but that's about it. Whatever order there -is is deduced from b-tree operator classes. - -The purpose of this patch is to raise collations to (reasonably) first -class object. The idea is that you can define a collation across any -type and that you will then be able to ORDER BY, GROUP BY and INDEX -using that collation. A collation defines both order and equality. - -The SQL standard does define COLLATE although they only apply that to -character strings. There are no predefined collations in the standard. -There are rules about how collations and collation states propegate -from the leaves of the parse tree all the way to the root. In its -simplest form, columns and constants have defined collations which -modify the behaviour of functions using these values. At any point in -the parse tree the user can override the collation with an explicit -. If there is ambiguity about what collation applies at -any point for a function that needs to know, this is a error. - -All this is parse-time analysis. - -Proposed Implementation: - -NODES - -To implement the above, two new node types are created: CollateClause -which represents the in SQL syntax, and CollateState -which represents the actual state at any node. Currently the only nodes -expected to require these are OpExpr, FuncExpr, Var and Const. Although -I guess it may apply to any node that can be used in an expression. - -CATALOG CHANGES - -To track collations requires a new table in the catalog, which I have -named pg_collations. It contains the following fields: - -Oid oid; -- OID for this collation -Name collname; -- Name of the collation (for collate clause) -bool collasc; -- Ascending or descending -Oid collopclass; -- Implementing Operator Class -int4 colltype; -- Currently, 0=3Dsimple, 1=3Duses locale -Oid colllocale; -- Locale in pg_locales - -(Should we be identifying the type here? or is it ok to lookup the type -via the operator class). - -The first few fields name the collation so it can be referred to by the -user. Then the collasc field determines how to use the operator class -as given in the collopclass field. If it indicates descending order, it -will invert the sense of the operator class. For example, asking for -the GT op for a reverse collation will actually return the LT operator -for the operator class. - -The purpose of the colltype and colllocale fields are described further -down. - -The important thing at this point is that by specifying a collation you -are also specifying an operator class. At the moment the ascending and -descending collations for each type are hard-coded for initdb. At the -moment they have been allocated OIDs starting at 2800, which is the -first large available block. - -Each column of a table has a default collation, which defaults to the -default collation of the type but can be specified in the table -declaration. To store this requires an additional column in -pg_attribute (attcollate) which contains the OID of the collation for -that column. When it is referenced in a query, this collation is copied -to the CollateState node of the Var node, from whence it can affect the -query. - -Finally, to allow the parser to complain about ambiguous CollateStates, -we need to indicate which functions actually need a senseble collate -state to function. To this end a single boolean field has been added to -pg_proc (proneedcollate). If this is true, the parser should error out -when the collation state is COLLATE_NONE. - -INDEXES - -Another place you will be allowed to use the collate clause is while -creating indexes. If you declare an index using a particular collation, -it can be used in queries that order by the same collation. Note, that -the collate clause indicates the operator class, so you can either -specify one or the other, but not both. - -So each column of an index will also have a collation. However, -pg_attribute has already got an extra field to store the collation for -columns so it makes sense to store the collation here. In the process -the pg_index.indclass field becomes redundant as it can be inferred -from the pg_attribute rows associated with the index. - -To make this work there also needs to be a notion of compatability -between collations. For example, two collations which are the reverse -of eachother are compatable in the sense that an index defined with one -collation would be usable for the other simply by scanning in reverse. - -FUNCTIONS - -In particular for string comparison but also possibly for user-defined -types, a function will need to know what collation it is operating -under. For this purpose an extra field (fn_collate) is added to -FmgrInfo which is filled in with the collation from the expression tree -(if any) or wherever relevent (eg. from the pg_attribute column when -doing statistics or creating indexes). - -A PG_GETCOLLATE() macro is added to facilitate user-functions -retreiving this data. This function throws an error when no collation -has been defined. This shouldn't happen in practice as issues should -have been weeded out at parse-time. - -This macro returns the OID of the collation but in many cases it will -not be necessary. In particular, functions should NOT invert their -result if the collation is inverted. It is considered the -responsibility of the caller to invert the result if necessary. The -reasons for this are: - -1. In most cases that matter (order comparison) the issue can be dealt -with at parse time by the NEGATOR or COMMUTATOR options. -2. For index scans, we would just do a reverse scan instead (or forward -if the index is inverted) -3. Requiring every function to check the collation for inversion is -wasteful, since in many cases the case can be dealt with statically. - -DEFAULT COLLATIONS - -At this point I'm inclined to define a few collations to be built in or -specially handled: - -COLLATE ASC - default collation for type, ascending (ie, what we do now) -COLLATE DESC - default collation for type, inverted -COLLATE POSIX - For strings, define a simple bytewise string comparison. - -Indeed, it is expected that by default, all columns involving strings -in the system catalog will always use COLLATE POSIX. Additionaly, type -"name" will always use that collation, even if the user changes the -default (by a method to be specified). This is straightforwardly done -at initdb time. - -The purpose of COLLATE DESC is to simplify index declarations. Saying - -CREATE INDEX foo ON bar( a COLLATE ASC, b COLLATE DESC ); - -would allow it to be used in a query using ORDER BY a, b DESC, without -the user having to lookup the name of the collation. - -When it comes to naming collations, the question arises whether -ascending/descending collations need to have different names. Or -should there be two collations with the same name with ASC/DESC as a -modifier? Do collations have to be unique across different types; for -example, can varchar and text both have a collation "ignorecase"? - -Another issue is that a column could be declared with a descending -collation by default. Say it was an integer column, then (a < 5) would -return FALSE for a =3D 1. While technically correct, I'm thinking of -ruling it out on the basis of being too confusing, and only allow -descending collations at query time or in index specifications. - -Another strange point at the moment is how to determine the default -collation of a type. At the moment it is done by finding the default -operator class and looking up the ascending version of that. However, -we may want to provide the user a way of specifying it directly, -perhaps by: - -ALTER TYPE text SET DEFAULT COLLATION ignorecase; - -PATHKEYS - -Currently during planning, pathkeys are indicated by an operator of the -operator class. Here we would simply replace that with the oid of the -collation, which can be matched directly with the collation defined by -the index. - -USER DEFINED TYPES - -None of this is interesting unless it can be applied to user-defined -types also. Fortunatly this is easy, when the user declares a b-tree -operator class, we can generate the collations automatically. We may -even allow the user to specify the name of the collation. However, if -the user wanted to create multiple collations based on the same -operator class (by using the PG_GETCOLLATE() macro above, we may want -to provide them a way of creating them directly. - -COLLATIONS USING LOCALES - -For strings, collation can be done in many different ways defined by -what is referred to as a locale. As indicated above in the definition -of pg_collations, there is a colltype field. For most collations this -will be 0 (simple collation). However, for strings the intention is to -use a type 1 (using locales). In this case the last column refers to -the OID of the locale, so you can many collations using the same -operator class, but different locale oids. On a system level it changes -nothing, but inside the functions implementing it, they should use -PG_GETLOCALE(). This will return an opaque pg_locale_t (see below) -handle which can then be used to implement the specifics. - -In principle, user-defined types need to be able to use this also, -perhaps by using the clause COLLATE USING LOCALE in the operator class. -In theory there should a collation for each combination of -locale-dependant datatype, locale and order ascending/descending. -How/when these are created has not yet be determined. - -MORE TYPES OF COLLATION - -Another collation type I've speculated about but not thought about -implementing is a "mapping collation", in which you map the values -through a function and then collate that. The obvious example would be -a case-insensetive mapping where lower is applied before collation. - -Implementation could be pretty much done by simply substituting the -functions into the parse tree. For example, if you defined something -like: - -CREATE COLLATION ignorecase ON text USING lower($1) COLLATE defaulttext; - -Then anytime you did a comparison with that collation, you would simply -insert those function calls into the parse-tree and then collate with -"defaulttext". When declaring an index you would just make it a -functional index. The rules for functional indexes should make it work -out-of-the-box. - -OTHER TECHNICAL ISSUES - -- Applying a COLLATE clause to an unknown literal causes it to be -coerced to the type that collation is based on. But what about if we -have something like COLLATE DESC? - -- This requires some changes in the bootstrap procedures given that we -need to be able to do lookups on the operator class for each type -fairly early on. Some are predefined but it does require moving the -opclass setup further up the list. However, if we store a default -collation in pg_type, we wouldn't need to do that. - -- Sorting arrays. Should they get their own collations, or should -they use the collations of their base types. - -LOCALES - -I've left this to the end because I don't want people distracted by -what is essentially a side-issue. While this would be needed to -implement COLLATE the way the SQL spec intended, it can actually be -implemented and dealt with as a seperate patch. The main reason a basic -implementation exists is that it provides a great way of finding places -that didn't define a collation, since any comparison involving "text" -requires one. - -To deal with locales I created another table in the catalog, -pg_locales. This provides an OID which can be referenced from -elsewhere, such as the pg_collations table. - -The design is intended to provide some pluggability, so locale -information can come from multiple sources. Also, each locale will be -referenced by an identifier which is unrelated to any external -identifier, so we're not bound by them. - -The columns defined currently are: - -Name locname - Identifier used by postgresql -Name locsysname - String identifying the locale for the locale provider -int4 locsource - System providing this locale -int4 locencoding - Encoding expected by provider - -It is expected that the list of sources for locale data will be short, -probably hard-coded into the backend (currenty internal/system/icu). -The only locale defined at startup is POSIX, which is implemented -internally. The intention is for any other locales to be defined at the -end of initdb. The expected syntax is something like: - -CREATE LOCALE hungarian AS 'hu_HU' USING glibc; - -This should use the provider to check the locale exists and has a -conpatible encoding. If so it is entered into the table ready for use. - -In the backend, there will be implementations of functions like -pg_strcoll_l, pg_localeconv_l, which work like the C system library -versions only they take an extra pg_locale_t argument. This is used to -dispatch the call to the right place. There will be a function to -quickly determine if a locale is C to shortcircuit complexity where it -is not needed. - -STATUS - -Implementation so far is available here: - -http://svana.org/kleptog/temp/collate-current.patch.gz - -This patch isn't "clean" and changes a few things that are not strictly -necessary. It won't finish initdb right now because it gets an error in -ANALYSE (the array issue above). - -Feedback, help, comments: please reply. - -Have a nice day, ---=20 -Martijn van Oosterhout http://svana.org/kleptog/ -> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a -> tool for doing 5% of the work and then sitting around waiting for someone -> else to do the other 95% so you can sue them. - ---W/nzBZO5zC0uMSeA -Content-Type: application/pgp-signature -Content-Disposition: inline - ------BEGIN PGP SIGNATURE----- -Version: GnuPG v1.0.6 (GNU/Linux) -Comment: For info see http://www.gnupg.org - -iD8DBQFDsUHDIB7bNG8LQkwRAnh0AJ0YUNLkVaSY3u0jWBPdlaq+9dujZACfTmis -JR1mF60lKx14Ih850p3lpVk= -=/Ghs ------END PGP SIGNATURE----- - ---W/nzBZO5zC0uMSeA-- - -From pgsql-hackers-owner+M78869@postgresql.org Tue Jan 24 10:59:03 2006 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id k0OGx0324883 - for ; Tue, 24 Jan 2006 11:59:00 -0500 (EST) -Received: from postgresql.org (postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 6E4BE67B09D; - Tue, 24 Jan 2006 12:58:56 -0400 (AST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by postgresql.org (Postfix) with ESMTP id 1C9E59DC9BE - for ; Tue, 24 Jan 2006 12:58:31 -0400 (AST) -Received: from postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 43415-09 - for ; - Tue, 24 Jan 2006 12:58:29 -0400 (AST) -Received: from svr4.postgresql.org (svr4.postgresql.org [66.98.251.159]) - by postgresql.org (Postfix) with ESMTP id F05519DC86E - for ; Tue, 24 Jan 2006 12:58:27 -0400 (AST) -Received: from alife.ru (alife.ru [82.146.44.110]) - by svr4.postgresql.org (Postfix) with ESMTP id 42D155AF193 - for ; Tue, 24 Jan 2006 16:58:27 +0000 (GMT) -Received: from [212.192.243.99] (elizabet.sai.msu.ru [212.192.243.99]) - (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) - (No client certificate requested) - by alife.ru (Postfix) with ESMTP id A54D554219D - for ; Tue, 24 Jan 2006 19:58:20 +0300 (MSK) -Message-ID: <43D65CA3.4000402@tronet.ru> -Date: Tue, 24 Jan 2006 19:58:11 +0300 -From: Alexey Slynko -User-Agent: Mozilla Thunderbird 1.0.7 (X11/20051005) -X-Accept-Language: ru-ru, ru -MIME-Version: 1.0 -To: pgsql-hackers@postgresql.org -Subject: [HACKERS] TODO item: locale per database patch (new iteration) -Content-Type: multipart/mixed; - boundary="------------060707070409000500060805" -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0 required=5 tests=[none] -X-Spam-Score: 0 -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -This is a multi-part message in MIME format. ---------------060707070409000500060805 -Content-Type: text/plain; charset=KOI8-R; format=flowed -Content-Transfer-Encoding: 7bit - -Hi, - -it's a renewed locale per database patch. Unfortunately, i've not found -clean way to rebuild database indexes automatically, if locale settings -of two databases (created and template) are differs. Now it's only -raises a NOTICE. So, if anyone has a right notion about it - let will -express. Comment and suggestions are highly appreciated - - ---------------060707070409000500060805 -Content-Type: text/plain; - name="locale_per_database.patch" -Content-Transfer-Encoding: 7bit -Content-Disposition: inline; - filename="locale_per_database.patch" - -Index: src/backend/access/transam/xlog.c -=================================================================== -RCS file: /projects/cvsroot/pgsql/src/backend/access/transam/xlog.c,v -retrieving revision 1.226 -diff -u -r1.226 xlog.c ---- src/backend/access/transam/xlog.c 11 Jan 2006 08:43:12 -0000 1.226 -+++ src/backend/access/transam/xlog.c 22 Jan 2006 16:41:02 -0000 -@@ -3394,7 +3394,6 @@ - { - int fd; - char buffer[BLCKSZ]; /* need not be aligned */ -- char *localeptr; - - /* - * Initialize version and compatibility-check fields -@@ -3418,18 +3417,6 @@ - ControlFile->enableIntTimes = FALSE; - #endif - -- ControlFile->localeBuflen = LOCALE_NAME_BUFLEN; -- localeptr = setlocale(LC_COLLATE, NULL); -- if (!localeptr) -- ereport(PANIC, -- (errmsg("invalid LC_COLLATE setting"))); -- StrNCpy(ControlFile->lc_collate, localeptr, LOCALE_NAME_BUFLEN); -- localeptr = setlocale(LC_CTYPE, NULL); -- if (!localeptr) -- ereport(PANIC, -- (errmsg("invalid LC_CTYPE setting"))); -- StrNCpy(ControlFile->lc_ctype, localeptr, LOCALE_NAME_BUFLEN); -- - /* Contents are protected with a CRC */ - INIT_CRC32(ControlFile->crc); - COMP_CRC32(ControlFile->crc, -@@ -3612,34 +3599,6 @@ - " but the server was compiled without HAVE_INT64_TIMESTAMP."), - errhint("It looks like you need to recompile or initdb."))); - #endif -- -- if (ControlFile->localeBuflen != LOCALE_NAME_BUFLEN) -- ereport(FATAL, -- (errmsg("database files are incompatible with server"), -- errdetail("The database cluster was initialized with LOCALE_NAME_BUFLEN %d," -- " but the server was compiled with LOCALE_NAME_BUFLEN %d.", -- ControlFile->localeBuflen, LOCALE_NAME_BUFLEN), -- errhint("It looks like you need to recompile or initdb."))); -- if (pg_perm_setlocale(LC_COLLATE, ControlFile->lc_collate) == NULL) -- ereport(FATAL, -- (errmsg("database files are incompatible with operating system"), -- errdetail("The database cluster was initialized with LC_COLLATE \"%s\"," -- " which is not recognized by setlocale().", -- ControlFile->lc_collate), -- errhint("It looks like you need to initdb or install locale support."))); -- if (pg_perm_setlocale(LC_CTYPE, ControlFile->lc_ctype) == NULL) -- ereport(FATAL, -- (errmsg("database files are incompatible with operating system"), -- errdetail("The database cluster was initialized with LC_CTYPE \"%s\"," -- " which is not recognized by setlocale().", -- ControlFile->lc_ctype), -- errhint("It looks like you need to initdb or install locale support."))); -- -- /* Make the fixed locale settings visible as GUC variables, too */ -- SetConfigOption("lc_collate", ControlFile->lc_collate, -- PGC_INTERNAL, PGC_S_OVERRIDE); -- SetConfigOption("lc_ctype", ControlFile->lc_ctype, -- PGC_INTERNAL, PGC_S_OVERRIDE); - } - - void -Index: src/backend/commands/dbcommands.c -=================================================================== -RCS file: /projects/cvsroot/pgsql/src/backend/commands/dbcommands.c,v -retrieving revision 1.175 -diff -u -r1.175 dbcommands.c ---- src/backend/commands/dbcommands.c 22 Nov 2005 18:17:08 -0000 1.175 -+++ src/backend/commands/dbcommands.c 22 Jan 2006 16:41:03 -0000 -@@ -25,6 +25,10 @@ - #include - #include - -+#ifdef HAVE_LANGINFO_H -+#include -+#endif -+ - #include "access/genam.h" - #include "access/heapam.h" - #include "catalog/catalog.h" -@@ -49,6 +53,7 @@ - #include "utils/fmgroids.h" - #include "utils/guc.h" - #include "utils/lsyscache.h" -+#include "utils/pg_locale.h" - #include "utils/syscache.h" - - -@@ -57,9 +62,11 @@ - int *encodingP, bool *dbIsTemplateP, bool *dbAllowConnP, - Oid *dbLastSysOidP, - TransactionId *dbVacuumXidP, TransactionId *dbFrozenXidP, -- Oid *dbTablespace); -+ Oid *dbTablespace, char **dbCollate, char **dbCtype); - static bool have_createdb_privilege(void); - static void remove_dbtablespaces(Oid db_id); -+static char * get_locale_encoding(const char *ctype); -+static int check_locale_encoding(int encid, const char *ctype); - - - /* -@@ -73,6 +80,8 @@ - Oid src_dboid; - Oid src_owner; - int src_encoding; -+ char *src_collate; -+ char *src_ctype; - bool src_istemplate; - bool src_allowconn; - Oid src_lastsysoid; -@@ -92,10 +101,14 @@ - DefElem *downer = NULL; - DefElem *dtemplate = NULL; - DefElem *dencoding = NULL; -+ DefElem *dlc_collate = NULL; -+ DefElem *dlc_ctype = NULL; - DefElem *dconnlimit = NULL; - char *dbname = stmt->dbname; - char *dbowner = NULL; - const char *dbtemplate = NULL; -+ char *lc_collate = NULL; -+ char *lc_ctype = NULL; - volatile int encoding = -1; - volatile int dbconnlimit = -1; - -@@ -139,6 +152,22 @@ - errmsg("conflicting or redundant options"))); - dencoding = defel; - } -+ else if (strcmp(defel->defname, "lccollate") == 0) -+ { -+ if (dlc_collate) -+ ereport(ERROR, -+ (errcode(ERRCODE_SYNTAX_ERROR), -+ errmsg("conflicting or redundant options"))); -+ dlc_collate = defel; -+ } -+ else if (strcmp(defel->defname, "lcctype") == 0) -+ { -+ if (dlc_ctype) -+ ereport(ERROR, -+ (errcode(ERRCODE_SYNTAX_ERROR), -+ errmsg("conflicting or redundant options"))); -+ dlc_ctype = defel; -+ } - else if (strcmp(defel->defname, "connectionlimit") == 0) - { - if (dconnlimit) -@@ -192,6 +221,22 @@ - elog(ERROR, "unrecognized node type: %d", - nodeTag(dencoding->arg)); - } -+ if (dlc_collate && dlc_collate->arg) { -+ lc_collate = strVal(dlc_collate->arg); -+ if ((locale_collate_assign(lc_collate, false, (GucSource)NULL)) == NULL) -+ ereport(ERROR, -+ (errcode(ERRCODE_UNDEFINED_OBJECT), -+ errmsg("%s is not a valid LC_COLLATE name", -+ lc_collate))); -+ } -+ if (dlc_ctype && dlc_ctype->arg) { -+ lc_ctype = strVal(dlc_ctype->arg); -+ if ((locale_collate_assign(lc_ctype, false, (GucSource)NULL)) == NULL) -+ ereport(ERROR, -+ (errcode(ERRCODE_UNDEFINED_OBJECT), -+ errmsg("%s is not a valid LC_CTYPE name", -+ lc_ctype))); -+ } - if (dconnlimit && dconnlimit->arg) - dbconnlimit = intVal(dconnlimit->arg); - -@@ -224,7 +269,7 @@ - * grab the exclusive lock. - */ - if (get_db_info(dbname, NULL, NULL, NULL, -- NULL, NULL, NULL, NULL, NULL, NULL)) -+ NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL)) - ereport(ERROR, - (errcode(ERRCODE_DUPLICATE_DATABASE), - errmsg("database \"%s\" already exists", dbname))); -@@ -237,7 +282,8 @@ - - if (!get_db_info(dbtemplate, &src_dboid, &src_owner, &src_encoding, - &src_istemplate, &src_allowconn, &src_lastsysoid, -- &src_vacuumxid, &src_frozenxid, &src_deftablespace)) -+ &src_vacuumxid, &src_frozenxid, &src_deftablespace, -+ &src_collate, &src_ctype)) - ereport(ERROR, - (errcode(ERRCODE_UNDEFINED_DATABASE), - errmsg("template database \"%s\" does not exist", dbtemplate))); -@@ -277,6 +323,21 @@ - (errcode(ERRCODE_WRONG_OBJECT_TYPE), - errmsg("invalid server encoding %d", encoding))); - -+ /* Set database lc_collate and lc_ctype */ -+ if (!lc_collate) -+ lc_collate = src_collate; -+ if (!lc_ctype) -+ lc_ctype = src_ctype; -+ -+#if defined(HAVE_LANGINFO_H) && defined(CODESET) -+ if (encoding > 0 && check_locale_encoding(encoding, lc_ctype) == -1) -+ ereport(ERROR, -+ (errcode(ERRCODE_WRONG_OBJECT_TYPE), -+ errmsg("encoding %s is not suitable for locale %s", -+ pg_encoding_to_char(encoding), -+ lc_ctype))); -+#endif -+ - /* Resolve default tablespace for new database */ - if (dtablespacename && dtablespacename->arg) - { -@@ -441,7 +502,7 @@ - - /* Check to see if someone else created same DB name meanwhile. */ - if (get_db_info(dbname, NULL, NULL, NULL, -- NULL, NULL, NULL, NULL, NULL, NULL)) -+ NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL)) - ereport(ERROR, - (errcode(ERRCODE_DUPLICATE_DATABASE), - errmsg("database \"%s\" already exists", dbname))); -@@ -459,6 +520,11 @@ - DirectFunctionCall1(namein, CStringGetDatum(dbname)); - new_record[Anum_pg_database_datdba - 1] = ObjectIdGetDatum(datdba); - new_record[Anum_pg_database_encoding - 1] = Int32GetDatum(encoding); -+ new_record[Anum_pg_database_datcollate - 1] = -+ DirectFunctionCall1(namein, CStringGetDatum(lc_collate)); -+ new_record[Anum_pg_database_datctype - 1] = -+ DirectFunctionCall1(namein, CStringGetDatum(lc_ctype)); -+ - new_record[Anum_pg_database_datistemplate - 1] = BoolGetDatum(false); - new_record[Anum_pg_database_datallowconn - 1] = BoolGetDatum(true); - new_record[Anum_pg_database_datconnlimit - 1] = Int32GetDatum(dbconnlimit); -@@ -527,6 +593,15 @@ - * Set flag to update flat database file at commit. - */ - database_file_update_needed(); -+ -+ /* -+ * Message about reindexing new database -+ */ -+ if (lc_collate != src_collate || lc_ctype != src_ctype) -+ ereport(NOTICE, -+ (errmsg("database \"%s\" need to be reindexed manually (REINDEX DATABASE)", -+ dbname))); -+ - } - PG_CATCH(); - { -@@ -584,7 +659,7 @@ - pgdbrel = heap_open(DatabaseRelationId, ExclusiveLock); - - if (!get_db_info(dbname, &db_id, NULL, NULL, -- &db_istemplate, NULL, NULL, NULL, NULL, NULL)) -+ &db_istemplate, NULL, NULL, NULL, NULL, NULL, NULL, NULL)) - { - if (!missing_ok) - { -@@ -1100,7 +1175,7 @@ - int *encodingP, bool *dbIsTemplateP, bool *dbAllowConnP, - Oid *dbLastSysOidP, - TransactionId *dbVacuumXidP, TransactionId *dbFrozenXidP, -- Oid *dbTablespace) -+ Oid *dbTablespace, char **dbCollate, char **dbCtype) - { - Relation relation; - ScanKeyData scanKey; -@@ -1155,6 +1230,11 @@ - /* default tablespace for this database */ - if (dbTablespace) - *dbTablespace = dbform->dattablespace; -+ /* default locale settings for this database */ -+ if (dbCollate) -+ *dbCollate = NameStr(dbform->datcollate); -+ if (dbCtype) -+ *dbCtype = NameStr(dbform->datctype); - } - - systable_endscan(scan); -@@ -1416,3 +1496,45 @@ - else - strcat(buf, "UNKNOWN"); - } -+ -+#if defined(HAVE_LANGINFO_H) && defined(CODESET) -+ -+static char * -+get_locale_encoding(const char *ctype) -+{ -+ char *save; -+ char *sys; -+ -+ save = setlocale(LC_CTYPE, NULL); -+ if (!save) -+ return NULL; -+ save = pstrdup(save); -+ -+ setlocale(LC_CTYPE, ctype); -+ sys = nl_langinfo(CODESET); -+ sys = pstrdup(sys); -+ -+ setlocale(LC_CTYPE, save); -+ pfree(save); -+ -+ return sys; -+} -+ -+static int -+check_locale_encoding(int encid, const char *ctype) -+{ -+ char *sys; -+ -+ sys = get_locale_encoding(ctype); -+ if (encid == pg_char_to_encoding(sys)) -+ { -+ pfree(sys); -+ return 0; -+ } -+ -+ pfree(sys); -+ return -1; -+} -+ -+#endif -+ -Index: src/backend/parser/gram.y -=================================================================== -RCS file: /projects/cvsroot/pgsql/src/backend/parser/gram.y,v -retrieving revision 2.522 -diff -u -r2.522 gram.y ---- src/backend/parser/gram.y 21 Jan 2006 02:16:19 -0000 2.522 -+++ src/backend/parser/gram.y 22 Jan 2006 16:41:09 -0000 -@@ -372,7 +372,7 @@ - - KEY - -- LANCOMPILER LANGUAGE LARGE_P LAST_P LEADING LEAST LEFT LEVEL -+ LANCOMPILER LANGUAGE LARGE_P LAST_P LCCOLLATE LCCTYPE LEADING LEAST LEFT LEVEL - LIKE LIMIT LISTEN LOAD LOCAL LOCALTIME LOCALTIMESTAMP LOCATION - LOCK_P LOGIN_P - -@@ -4635,6 +4635,22 @@ - { - $$ = makeDefElem("encoding", NULL); - } -+ | LCCOLLATE opt_equal name -+ { -+ $$ = makeDefElem("lccollate", (Node *)makeString($3)); -+ } -+ | LCCOLLATE opt_equal DEFAULT -+ { -+ $$ = makeDefElem("lccollate", NULL); -+ } -+ | LCCTYPE opt_equal name -+ { -+ $$ = makeDefElem("lcctype", (Node *)makeString($3)); -+ } -+ | LCCTYPE opt_equal DEFAULT -+ { -+ $$ = makeDefElem("lcctype", NULL); -+ } - | CONNECTION LIMIT opt_equal SignedIconst - { - $$ = makeDefElem("connectionlimit", (Node *)makeInteger($4)); -@@ -8225,6 +8241,8 @@ - | LANGUAGE - | LARGE_P - | LAST_P -+ | LCCOLLATE -+ | LCCTYPE - | LEVEL - | LISTEN - | LOAD -Index: src/backend/parser/keywords.c -=================================================================== -RCS file: /projects/cvsroot/pgsql/src/backend/parser/keywords.c,v -retrieving revision 1.170 -diff -u -r1.170 keywords.c ---- src/backend/parser/keywords.c 27 Dec 2005 04:00:07 -0000 1.170 -+++ src/backend/parser/keywords.c 22 Jan 2006 16:41:09 -0000 -@@ -193,6 +193,8 @@ - {"language", LANGUAGE}, - {"large", LARGE_P}, - {"last", LAST_P}, -+ {"lccollate", LCCOLLATE}, -+ {"lcctype", LCCTYPE}, - {"leading", LEADING}, - {"least", LEAST}, - {"left", LEFT}, -Index: src/backend/utils/adt/pg_locale.c -=================================================================== -RCS file: /projects/cvsroot/pgsql/src/backend/utils/adt/pg_locale.c,v -retrieving revision 1.34 -diff -u -r1.34 pg_locale.c ---- src/backend/utils/adt/pg_locale.c 2 Jan 2006 20:25:45 -0000 1.34 -+++ src/backend/utils/adt/pg_locale.c 22 Jan 2006 16:41:10 -0000 -@@ -10,10 +10,8 @@ - */ - - /*---------- -- * Here is how the locale stuff is handled: LC_COLLATE and LC_CTYPE -- * are fixed by initdb, stored in pg_control, and cannot be changed. -- * Thus, the effects of strcoll(), strxfrm(), isupper(), toupper(), -- * etc. are always in the same fixed locale. -+ * Here is how the locale stuff is handled: -+ * LC_COLLATE and LC_CTYPE are defined by createdb and stored in pg_database. - * - * LC_MESSAGES is settable at run time and will take effect - * immediately. -@@ -208,6 +206,17 @@ - return value; - } - -+const char * -+locale_collate_assign(const char *value, bool doit, GucSource source) -+{ -+ return locale_xxx_assign(LC_COLLATE, value, doit, source); -+} -+ -+const char * -+locale_ctype_assign(const char *value, bool doit, GucSource source) -+{ -+ return locale_xxx_assign(LC_CTYPE, value, doit, source); -+} - - const char * - locale_monetary_assign(const char *value, bool doit, GucSource source) -Index: src/backend/utils/init/postinit.c -=================================================================== -RCS file: /projects/cvsroot/pgsql/src/backend/utils/init/postinit.c,v -retrieving revision 1.160 -diff -u -r1.160 postinit.c ---- src/backend/utils/init/postinit.c 4 Jan 2006 21:06:32 -0000 1.160 -+++ src/backend/utils/init/postinit.c 22 Jan 2006 16:41:10 -0000 -@@ -138,6 +138,8 @@ - ScanKeyData key; - HeapTuple tup; - Form_pg_database dbform; -+ char *lc_ctype; -+ char *lc_collate; - - /* - * Because we grab RowShareLock here, we can be sure that dropdb() is not -@@ -225,6 +227,32 @@ - SetConfigOption("client_encoding", GetDatabaseEncodingName(), - PGC_BACKEND, PGC_S_DEFAULT); - -+ /* Set up database locale */ -+ lc_collate = NameStr(dbform->datcollate); -+ lc_ctype = NameStr(dbform->datctype); -+ -+ if (setlocale(LC_COLLATE, lc_collate) == NULL) -+ ereport(FATAL, -+ (errmsg("database locale is incompatible with operating system"), -+ errdetail("The database was initialized with LC_COLLATE \"%s\"," -+ " which is not recognized by setlocale().", -+ lc_collate), -+ errhint("It looks like you need to recreate database or install locale support."))); -+ if (setlocale(LC_CTYPE, lc_ctype) == NULL) -+ ereport(FATAL, -+ (errmsg("database locale are incompatible with operating system"), -+ errdetail("The database was initialized with LC_CTYPE \"%s\"," -+ " which is not recognized by setlocale().", -+ lc_ctype), -+ errhint("It looks like you need to recreate database or install locale support."))); -+ -+ /* Record it as a GUC internal option, too */ -+ SetConfigOption("lc_collate", lc_collate, -+ PGC_INTERNAL, PGC_S_DATABASE); -+ SetConfigOption("lc_ctype", lc_ctype, -+ PGC_INTERNAL, PGC_S_DATABASE); -+ -+ - /* - * Lastly, set up any database-specific configuration variables. - */ -Index: src/bin/initdb/initdb.c -=================================================================== -RCS file: /projects/cvsroot/pgsql/src/bin/initdb/initdb.c,v -retrieving revision 1.106 -diff -u -r1.106 initdb.c ---- src/bin/initdb/initdb.c 5 Jan 2006 10:07:46 -0000 1.106 -+++ src/bin/initdb/initdb.c 22 Jan 2006 16:41:13 -0000 -@@ -1377,6 +1377,10 @@ - - bki_lines = replace_token(bki_lines, "ENCODING", encodingid); - -+ bki_lines = replace_token(bki_lines, "LC_COLLATE", lc_collate); -+ -+ bki_lines = replace_token(bki_lines, "LC_CTYPE", lc_ctype); -+ - /* - * Pass correct LC_xxx environment to bootstrap. - * -@@ -2617,7 +2621,7 @@ - printf(_("The database cluster will be initialized with locale %s.\n"), lc_ctype); - else - { -- printf(_("The database cluster will be initialized with locales\n" -+ printf(_("The database template1 will be initialized with locales\n" - " COLLATE: %s\n" - " CTYPE: %s\n" - " MESSAGES: %s\n" -Index: src/bin/pg_controldata/pg_controldata.c -=================================================================== -RCS file: /projects/cvsroot/pgsql/src/bin/pg_controldata/pg_controldata.c,v -retrieving revision 1.27 -diff -u -r1.27 pg_controldata.c ---- src/bin/pg_controldata/pg_controldata.c 15 Oct 2005 02:49:37 -0000 1.27 -+++ src/bin/pg_controldata/pg_controldata.c 22 Jan 2006 16:41:13 -0000 -@@ -177,9 +177,5 @@ - printf(_("Maximum columns in an index: %u\n"), ControlFile.indexMaxKeys); - printf(_("Date/time type storage: %s\n"), - (ControlFile.enableIntTimes ? _("64-bit integers") : _("floating-point numbers"))); -- printf(_("Maximum length of locale name: %u\n"), ControlFile.localeBuflen); -- printf(_("LC_COLLATE: %s\n"), ControlFile.lc_collate); -- printf(_("LC_CTYPE: %s\n"), ControlFile.lc_ctype); -- - return 0; - } -Index: src/bin/pg_resetxlog/pg_resetxlog.c -=================================================================== -RCS file: /projects/cvsroot/pgsql/src/bin/pg_resetxlog/pg_resetxlog.c,v -retrieving revision 1.39 -diff -u -r1.39 pg_resetxlog.c ---- src/bin/pg_resetxlog/pg_resetxlog.c 5 Jan 2006 03:01:37 -0000 1.39 -+++ src/bin/pg_resetxlog/pg_resetxlog.c 22 Jan 2006 16:41:13 -0000 -@@ -465,22 +465,6 @@ - #else - ControlFile.enableIntTimes = FALSE; - #endif -- ControlFile.localeBuflen = LOCALE_NAME_BUFLEN; -- -- localeptr = setlocale(LC_COLLATE, ""); -- if (!localeptr) -- { -- fprintf(stderr, _("%s: invalid LC_COLLATE setting\n"), progname); -- exit(1); -- } -- StrNCpy(ControlFile.lc_collate, localeptr, LOCALE_NAME_BUFLEN); -- localeptr = setlocale(LC_CTYPE, ""); -- if (!localeptr) -- { -- fprintf(stderr, _("%s: invalid LC_CTYPE setting\n"), progname); -- exit(1); -- } -- StrNCpy(ControlFile.lc_ctype, localeptr, LOCALE_NAME_BUFLEN); - - /* - * XXX eventually, should try to grovel through old XLOG to develop more -@@ -530,9 +514,6 @@ - printf(_("Maximum columns in an index: %u\n"), ControlFile.indexMaxKeys); - printf(_("Date/time type storage: %s\n"), - (ControlFile.enableIntTimes ? _("64-bit integers") : _("floating-point numbers"))); -- printf(_("Maximum length of locale name: %u\n"), ControlFile.localeBuflen); -- printf(_("LC_COLLATE: %s\n"), ControlFile.lc_collate); -- printf(_("LC_CTYPE: %s\n"), ControlFile.lc_ctype); - } - - -Index: src/bin/psql/describe.c -=================================================================== -RCS file: /projects/cvsroot/pgsql/src/bin/psql/describe.c,v -retrieving revision 1.130 -diff -u -r1.130 describe.c ---- src/bin/psql/describe.c 22 Nov 2005 18:17:29 -0000 1.130 -+++ src/bin/psql/describe.c 22 Jan 2006 16:41:15 -0000 -@@ -360,6 +360,12 @@ - appendPQExpBuffer(&buf, - ",\n pg_catalog.pg_encoding_to_char(d.encoding) as \"%s\"", - _("Encoding")); -+ appendPQExpBuffer(&buf, -+ ",\n d.datcollate as \"%s\"", -+ _("LC_COLLATE")); -+ appendPQExpBuffer(&buf, -+ ",\n d.datctype as \"%s\"", -+ _("LC_CTYPE")); - if (verbose) - appendPQExpBuffer(&buf, - ",\n pg_catalog.obj_description(d.oid, 'pg_database') as \"%s\"", -Index: src/bin/scripts/createdb.c -=================================================================== -RCS file: /projects/cvsroot/pgsql/src/bin/scripts/createdb.c,v -retrieving revision 1.15 -diff -u -r1.15 createdb.c ---- src/bin/scripts/createdb.c 21 Jun 2005 04:02:33 -0000 1.15 -+++ src/bin/scripts/createdb.c 22 Jan 2006 16:41:15 -0000 -@@ -34,6 +34,8 @@ - {"tablespace", required_argument, NULL, 'D'}, - {"template", required_argument, NULL, 'T'}, - {"encoding", required_argument, NULL, 'E'}, -+ {"lc-collate", required_argument, NULL, 1}, -+ {"lc-ctype", required_argument, NULL, 2}, - {NULL, 0, NULL, 0} - }; - -@@ -53,6 +55,8 @@ - char *tablespace = NULL; - char *template = NULL; - char *encoding = NULL; -+ char *lc_collate = NULL; -+ char *lc_ctype = NULL; - - PQExpBufferData sql; - -@@ -98,6 +102,12 @@ - case 'E': - encoding = optarg; - break; -+ case 1: -+ lc_collate = optarg; -+ break; -+ case 2: -+ lc_ctype = optarg; -+ break; - default: - fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname); - exit(1); -@@ -155,7 +165,12 @@ - appendPQExpBuffer(&sql, " ENCODING '%s'", encoding); - if (template) - appendPQExpBuffer(&sql, " TEMPLATE %s", fmtId(template)); -+ if (lc_collate) -+ appendPQExpBuffer(&sql, " LCCOLLATE %s", fmtId(lc_collate)); -+ if (lc_ctype) -+ appendPQExpBuffer(&sql, " LCCTYPE %s", fmtId(lc_ctype)); - appendPQExpBuffer(&sql, ";\n"); -+ - - conn = connectDatabase(strcmp(dbname, "postgres") == 0 ? "template1" : "postgres", - host, port, username, password, progname); -@@ -219,19 +234,20 @@ - printf(_("Usage:\n")); - printf(_(" %s [OPTION]... [DBNAME] [DESCRIPTION]\n"), progname); - printf(_("\nOptions:\n")); -- printf(_(" -D, --tablespace=TABLESPACE default tablespace for the database\n")); -- printf(_(" -E, --encoding=ENCODING encoding for the database\n")); -- printf(_(" -O, --owner=OWNER database user to own the new database\n")); -- printf(_(" -T, --template=TEMPLATE template database to copy\n")); -- printf(_(" -e, --echo show the commands being sent to the server\n")); -- printf(_(" -q, --quiet don't write any messages\n")); -- printf(_(" --help show this help, then exit\n")); -- printf(_(" --version output version information, then exit\n")); -+ printf(_(" -D, --tablespace=TABLESPACE default tablespace for the database\n")); -+ printf(_(" -E, --encoding=ENCODING encoding for the database\n")); -+ printf(_(" --lc-collate, --lc-ctype=LOCALE initialize database with given locale\n")); -+ printf(_(" -O, --owner=OWNER database user to own the new database\n")); -+ printf(_(" -T, --template=TEMPLATE template database to copy\n")); -+ printf(_(" -e, --echo show the commands being sent to the server\n")); -+ printf(_(" -q, --quiet don't write any messages\n")); -+ printf(_(" --help show this help, then exit\n")); -+ printf(_(" --version output version information, then exit\n")); - printf(_("\nConnection options:\n")); -- printf(_(" -h, --host=HOSTNAME database server host or socket directory\n")); -- printf(_(" -p, --port=PORT database server port\n")); -- printf(_(" -U, --username=USERNAME user name to connect as\n")); -- printf(_(" -W, --password prompt for password\n")); -+ printf(_(" -h, --host=HOSTNAME database server host or socket directory\n")); -+ printf(_(" -p, --port=PORT database server port\n")); -+ printf(_(" -U, --username=USERNAME user name to connect as\n")); -+ printf(_(" -W, --password prompt for password\n")); - printf(_("\nBy default, a database with the same name as the current user is created.\n")); - printf(_("\nReport bugs to .\n")); - } -Index: src/include/catalog/pg_control.h -=================================================================== -RCS file: /projects/cvsroot/pgsql/src/include/catalog/pg_control.h,v -retrieving revision 1.26 -diff -u -r1.26 pg_control.h ---- src/include/catalog/pg_control.h 22 Nov 2005 18:17:30 -0000 1.26 -+++ src/include/catalog/pg_control.h 22 Jan 2006 16:41:15 -0000 -@@ -137,11 +137,6 @@ - /* flag indicating internal format of timestamp, interval, time */ - uint32 enableIntTimes; /* int64 storage enabled? */ - -- /* active locales */ -- uint32 localeBuflen; -- char lc_collate[LOCALE_NAME_BUFLEN]; -- char lc_ctype[LOCALE_NAME_BUFLEN]; -- - /* CRC of all above ... MUST BE LAST! */ - pg_crc32 crc; - } ControlFileData; -Index: src/include/catalog/pg_database.h -=================================================================== -RCS file: /projects/cvsroot/pgsql/src/include/catalog/pg_database.h,v -retrieving revision 1.38 -diff -u -r1.38 pg_database.h ---- src/include/catalog/pg_database.h 15 Oct 2005 02:49:42 -0000 1.38 -+++ src/include/catalog/pg_database.h 22 Jan 2006 16:41:16 -0000 -@@ -38,6 +38,8 @@ - NameData datname; /* database name */ - Oid datdba; /* owner of database */ - int4 encoding; /* character encoding */ -+ NameData datcollate; /* locale LC_COLLATE */ -+ NameData datctype; /* locale LC_CTYPE */ - bool datistemplate; /* allowed as CREATE DATABASE template? */ - bool datallowconn; /* new connections allowed? */ - int4 datconnlimit; /* max connections allowed (-1=no limit) */ -@@ -60,21 +62,23 @@ - * compiler constants for pg_database - * ---------------- - */ --#define Natts_pg_database 12 -+#define Natts_pg_database 14 - #define Anum_pg_database_datname 1 - #define Anum_pg_database_datdba 2 - #define Anum_pg_database_encoding 3 --#define Anum_pg_database_datistemplate 4 --#define Anum_pg_database_datallowconn 5 --#define Anum_pg_database_datconnlimit 6 --#define Anum_pg_database_datlastsysoid 7 --#define Anum_pg_database_datvacuumxid 8 --#define Anum_pg_database_datfrozenxid 9 --#define Anum_pg_database_dattablespace 10 --#define Anum_pg_database_datconfig 11 --#define Anum_pg_database_datacl 12 -+#define Anum_pg_database_datcollate 4 -+#define Anum_pg_database_datctype 5 -+#define Anum_pg_database_datistemplate 6 -+#define Anum_pg_database_datallowconn 7 -+#define Anum_pg_database_datconnlimit 8 -+#define Anum_pg_database_datlastsysoid 9 -+#define Anum_pg_database_datvacuumxid 10 -+#define Anum_pg_database_datfrozenxid 11 -+#define Anum_pg_database_dattablespace 12 -+#define Anum_pg_database_datconfig 13 -+#define Anum_pg_database_datacl 14 - --DATA(insert OID = 1 ( template1 PGUID ENCODING t t -1 0 0 0 1663 _null_ _null_ )); -+DATA(insert OID = 1 ( template1 PGUID ENCODING "LC_CTYPE" "LC_COLLATE" t t -1 0 0 0 1663 _null_ _null_ )); - DESCR("Default template database"); - #define TemplateDbOid 1 - -Index: src/include/utils/pg_locale.h -=================================================================== -RCS file: /projects/cvsroot/pgsql/src/include/utils/pg_locale.h,v -retrieving revision 1.21 -diff -u -r1.21 pg_locale.h ---- src/include/utils/pg_locale.h 28 Dec 2005 23:22:51 -0000 1.21 -+++ src/include/utils/pg_locale.h 22 Jan 2006 16:41:16 -0000 -@@ -22,6 +22,10 @@ - extern char *locale_numeric; - extern char *locale_time; - -+extern const char *locale_collate_assign(const char *value, -+ bool doit, GucSource source); -+extern const char *locale_ctype_assign(const char *value, -+ bool doit, GucSource source); - extern const char *locale_messages_assign(const char *value, - bool doit, GucSource source); - extern const char *locale_monetary_assign(const char *value, - - ---------------060707070409000500060805 -Content-Type: text/plain -Content-Disposition: inline -Content-Transfer-Encoding: 8bit -MIME-Version: 1.0 - - ----------------------------(end of broadcast)--------------------------- -TIP 1: if posting/reading through Usenet, please send an appropriate - subscribe-nomail command to majordomo@postgresql.org so that your - message can get through to the mailing list cleanly - ---------------060707070409000500060805-- - diff --git a/doc/TODO.detail/merge b/doc/TODO.detail/merge deleted file mode 100644 index 276e2528a3..0000000000 --- a/doc/TODO.detail/merge +++ /dev/null @@ -1,3390 +0,0 @@ -From kleptog@svana.org Thu Nov 17 07:33:25 2005 -Return-path: -Received: from svana.org (mail@svana.org [203.20.62.76]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jAHCXNu09324 - for ; Thu, 17 Nov 2005 07:33:25 -0500 (EST) -Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian)) - id 1Eciwk-0007Gd-00; Thu, 17 Nov 2005 23:32:58 +1100 -Date: Thu, 17 Nov 2005 13:32:57 +0100 -From: Martijn van Oosterhout -To: Csaba Nagy -cc: Zeugswetter Andreas DCP SD , - Dann Corbit , Simon Riggs , - Bruce Momjian , - Rick Gigger , Tom Lane , - Christopher Kings-Lynne , - "Jim C. Nasby" , josh@agliodbs.com, - pgsql-hackers@postgresql.org, Jaime Casanova , - Peter Eisentraut -Subject: Re: [HACKERS] MERGE vs REPLACE -Message-ID: <20051117123250.GC22933@svana.org> -Reply-To: Martijn van Oosterhout -References: <1132228373.10890.313.camel@coppola.muc.ecircle.de> -MIME-Version: 1.0 -Content-Type: multipart/signed; micalg=pgp-sha1; - protocol="application/pgp-signature"; boundary="raC6veAxrt5nqIoY" -Content-Disposition: inline -In-Reply-To: <1132228373.10890.313.camel@coppola.muc.ecircle.de> -User-Agent: Mutt/1.3.28i -X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6 -X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522 48A7 6394 F08A 0DC6 7BE6 -X-PGP-Key-URL: -Status: OR - - ---raC6veAxrt5nqIoY -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -Content-Transfer-Encoding: quoted-printable - -On Thu, Nov 17, 2005 at 12:52:53PM +0100, Csaba Nagy wrote: -> Yes, these algorithms are clear to me, but they don't work for batch -> updates in postgres without savepoints before each row insert/update, -> which is not good for performance (not to mention on older postgres -> versions without savepoint support it won't work at all). If there is a -> way of no race condition, no performance penalty, that would be -> something new and useful. I just guess the MERGE would provide that. - -Well, then you guess wrong. This isn't what MERGE is for. MERGE is just -a neat way of specifying the UPDATE and INSERT cases in the same -statement. It doesn't remove the possibility duplicate inserts and thus -primary key violations. - -If someone wants to make extensions to MERGE so that it can avoid the -race condition and avoid the duplicate key violations, that's fine. But -be aware that this is outside of the spec. It may be a useful addition, -but perhaps we should consider MERGE and REPLACE as completely seperate -targets. - -MERGE has a whole join construction with subqueries that would be a -pain to make work in a way that is truly serialisable. REPLACE deals -with only one row and tries to solve the race for that case only. Much -easier to consider them seperately, no? - -I guess what's really irritating is that this clearly exposes the case -listed in the docs as "Why SERIALIZABLE isn't in all cases". If we -could solve that for MERGE, we could probably solve it in the general -case too. - -Have a nice day, ---=20 -Martijn van Oosterhout http://svana.org/kleptog/ -> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a -> tool for doing 5% of the work and then sitting around waiting for someone -> else to do the other 95% so you can sue them. - ---raC6veAxrt5nqIoY -Content-Type: application/pgp-signature -Content-Disposition: inline - ------BEGIN PGP SIGNATURE----- -Version: GnuPG v1.0.6 (GNU/Linux) -Comment: For info see http://www.gnupg.org - -iD8DBQFDfHhxIB7bNG8LQkwRAhIwAJwPmzE2GHrqzPujkkj2I5r6OlVo5QCeN4st -Ka50Vh0AnXuj4pBt27V6j7I= -=7rb7 ------END PGP SIGNATURE----- - ---raC6veAxrt5nqIoY-- - -From pgsql-hackers-owner+M76186@postgresql.org Thu Nov 17 09:44:46 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jAHEihu07081 - for ; Thu, 17 Nov 2005 09:44:44 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 84700C4B33D; - Thu, 17 Nov 2005 14:44:42 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 84C6BD967F - for ; Thu, 17 Nov 2005 10:41:57 -0400 (AST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 47040-01 - for ; - Thu, 17 Nov 2005 14:42:00 +0000 (GMT) -X-Greylist: from auto-whitelisted by SQLgrey- -Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) - by svr1.postgresql.org (Postfix) with ESMTP id ECB3CD963C - for ; Thu, 17 Nov 2005 10:41:53 -0400 (AST) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id jAHEfAuK025850; - Thu, 17 Nov 2005 09:41:11 -0500 (EST) -To: Csaba Nagy -cc: Martijn van Oosterhout , - Zeugswetter Andreas DCP SD , - Dann Corbit , Simon Riggs , - Bruce Momjian , - Rick Gigger , - Christopher Kings-Lynne , - "Jim C. Nasby" , josh@agliodbs.com, - pgsql-hackers@postgresql.org, Jaime Casanova , - Peter Eisentraut -Subject: Re: [HACKERS] MERGE vs REPLACE -In-Reply-To: <1132231474.10890.317.camel@coppola.muc.ecircle.de> -References: <1132228373.10890.313.camel@coppola.muc.ecircle.de> <20051117123250.GC22933@svana.org> <1132231474.10890.317.camel@coppola.muc.ecircle.de> -Comments: In-reply-to Csaba Nagy - message dated "Thu, 17 Nov 2005 13:44:35 +0100" -Date: Thu, 17 Nov 2005 09:41:10 -0500 -Message-ID: <25849.1132238470@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.008 required=5 tests=[AWL=0.008] -X-Spam-Score: 0.008 -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: ORr - -Csaba Nagy writes: -> OK, in this case I don't care about either MERGE or REPLACE, but for an -> UPSERT which does the locking :-) - -This is exactly the point --- pretty much nobody has come to us and -asked for a feature that does what Peter and Martijn say MERGE does. -(I haven't bothered to look at the 2003 spec, I'm assuming they read it -correctly.) What we *have* been asked for, over and over, is an -insert-or-update feature that's not so tedious and inefficient as the -savepoint-insert-rollback-update kluge. That's what we ought to be -concentrating on providing. - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 3: Have you checked our extensive FAQ? - - http://www.postgresql.org/docs/faq - -From pgsql-hackers-owner+M76189@postgresql.org Thu Nov 17 10:17:57 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jAHFHvu12126 - for ; Thu, 17 Nov 2005 10:17:57 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 9B0AFC4B33D; - Thu, 17 Nov 2005 15:17:55 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 02FEFDB988 - for ; Thu, 17 Nov 2005 11:14:52 -0400 (AST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 48136-10 - for ; - Thu, 17 Nov 2005 15:14:56 +0000 (GMT) -X-Greylist: from auto-whitelisted by SQLgrey- -Received: from svr2.postgresql.org (svr2.postgresql.org [65.19.161.25]) - by svr1.postgresql.org (Postfix) with ESMTP id 34BD1DB97C - for ; Thu, 17 Nov 2005 11:14:50 -0400 (AST) -Received: from ns.snowman.net (ns.snowman.net [66.92.160.21]) - by svr2.postgresql.org (Postfix) with ESMTP id A8A8FF0BEF - for ; Thu, 17 Nov 2005 15:14:54 +0000 (GMT) -Received: by ns.snowman.net (Postfix, from userid 1000) - id EDB5717AD6; Thu, 17 Nov 2005 10:15:30 -0500 (EST) -Date: Thu, 17 Nov 2005 10:15:30 -0500 -From: Stephen Frost -To: Tom Lane -cc: Csaba Nagy , - Martijn van Oosterhout , - Zeugswetter Andreas DCP SD , - Dann Corbit , Simon Riggs , - Bruce Momjian , - Rick Gigger , - Christopher Kings-Lynne , - "Jim C. Nasby" , josh@agliodbs.com, - pgsql-hackers@postgresql.org, Jaime Casanova , - Peter Eisentraut -Subject: Re: [HACKERS] MERGE vs REPLACE -Message-ID: <20051117151530.GU6026@ns.snowman.net> -Mail-Followup-To: Tom Lane , - Csaba Nagy , - Martijn van Oosterhout , - Zeugswetter Andreas DCP SD , - Dann Corbit , Simon Riggs , - Bruce Momjian , - Rick Gigger , - Christopher Kings-Lynne , - "Jim C. Nasby" , josh@agliodbs.com, - pgsql-hackers@postgresql.org, - Jaime Casanova , - Peter Eisentraut -References: <1132228373.10890.313.camel@coppola.muc.ecircle.de> <20051117123250.GC22933@svana.org> <1132231474.10890.317.camel@coppola.muc.ecircle.de> <25849.1132238470@sss.pgh.pa.us> -MIME-Version: 1.0 -Content-Type: multipart/signed; micalg=pgp-sha1; - protocol="application/pgp-signature"; boundary="aAYbr14jHAy2Yyau" -Content-Disposition: inline -In-Reply-To: <25849.1132238470@sss.pgh.pa.us> -X-Editor: Vim http://www.vim.org/ -X-Info: http://www.snowman.net -X-Operating-System: Linux/2.4.24ns.3.0 (i686) -X-Uptime: 10:01:46 up 159 days, 7:15, 2 users, load average: 0.31, 0.23, 0.13 -User-Agent: Mutt/1.5.9i -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0 required=5 tests=[none] -X-Spam-Score: 0 -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - ---aAYbr14jHAy2Yyau -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -Content-Transfer-Encoding: quoted-printable - -* Tom Lane (tgl@sss.pgh.pa.us) wrote: -> This is exactly the point --- pretty much nobody has come to us and -> asked for a feature that does what Peter and Martijn say MERGE does. -> (I haven't bothered to look at the 2003 spec, I'm assuming they read it -> correctly.) What we *have* been asked for, over and over, is an -> insert-or-update feature that's not so tedious and inefficient as the -> savepoint-insert-rollback-update kluge. That's what we ought to be -> concentrating on providing. - -I guess to be clear on what this distinction actually is, specifically: -MERGE under SQL2003 doesn't appear to be intended to be used -concurrently. For data warehousing situations this can be just fine -such as in my case where I get a monthly update of some information and -need to merge that update in with the prior information. In this case -there's only one MERGE running and I'd hope it'd be faster than doing -check for existance, insert/update on each row in plpgsql or something -(since there'd be multiple index lookups, etc, I think). Concurrent -MERGEs running *can* fail, just like whole transactions which do the -check/insert/update can fail. - -REPLACE/INSERT ON DUPLICATE UPDATE appears to essentially be a -transaction which is supposed to not fail but instead do locking to -ensure that it doesn't fail. This requires predicate locking to be -efficient because you want to tell the concurrent transaction "if you -have the same key as me, just wait a second and you can do an update -'cause I'm going to create the key if it doesn't exist before I'm done". - -I think REPLACE/INSERT ON DUPLICATE UPDATE is definitely harder to do -than MERGE because of the idea that it isn't supposed to fail generally. -I think SQL2003 MERGE would be reasonably easy to do and to get the -efficiency benefits out of it (assuming there are some to be had in the -end). =20 - -I don't think MERGE can really be made to be both though, in which case -it should really be the SQL2003 MERGE and we can make REPLACE/INSERT ON -DUPLICATE UPDATE something else. Perhaps a special form of MERGE where -you know it's going to be doing that locking. I really don't like the -idea of making the SQL2003 version of MERGE be the MERGE special case -(by requiring someone to take a table lock ahead of time or do something -else odd). - - Thanks, - - Stephen - ---aAYbr14jHAy2Yyau -Content-Type: application/pgp-signature; name="signature.asc" -Content-Description: Digital signature -Content-Disposition: inline - ------BEGIN PGP SIGNATURE----- -Version: GnuPG v1.4.2 (GNU/Linux) - -iD8DBQFDfJ6SrzgMPqB3kigRAjXWAJ9R/50PoocURxvi74g7dwhIO4akgQCcDEDG -4hGZAVR/9Age8pFtEOp4kfo= -=F91e ------END PGP SIGNATURE----- - ---aAYbr14jHAy2Yyau-- - -From sfrost@snowman.net Thu Nov 17 10:14:59 2005 -Return-path: -Received: from ns.snowman.net (ns.snowman.net [66.92.160.21]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jAHFEwu11635 - for ; Thu, 17 Nov 2005 10:14:58 -0500 (EST) -Received: by ns.snowman.net (Postfix, from userid 1000) - id EDB5717AD6; Thu, 17 Nov 2005 10:15:30 -0500 (EST) -Date: Thu, 17 Nov 2005 10:15:30 -0500 -From: Stephen Frost -To: Tom Lane -cc: Csaba Nagy , - Martijn van Oosterhout , - Zeugswetter Andreas DCP SD , - Dann Corbit , Simon Riggs , - Bruce Momjian , - Rick Gigger , - Christopher Kings-Lynne , - "Jim C. Nasby" , josh@agliodbs.com, - pgsql-hackers@postgresql.org, Jaime Casanova , - Peter Eisentraut -Subject: Re: [HACKERS] MERGE vs REPLACE -Message-ID: <20051117151530.GU6026@ns.snowman.net> -Mail-Followup-To: Tom Lane , - Csaba Nagy , - Martijn van Oosterhout , - Zeugswetter Andreas DCP SD , - Dann Corbit , - Simon Riggs , - Bruce Momjian , - Rick Gigger , - Christopher Kings-Lynne , - "Jim C. Nasby" , josh@agliodbs.com, - pgsql-hackers@postgresql.org, - Jaime Casanova , - Peter Eisentraut -References: <1132228373.10890.313.camel@coppola.muc.ecircle.de> <20051117123250.GC22933@svana.org> <1132231474.10890.317.camel@coppola.muc.ecircle.de> <25849.1132238470@sss.pgh.pa.us> -MIME-Version: 1.0 -Content-Type: multipart/signed; micalg=pgp-sha1; - protocol="application/pgp-signature"; boundary="aAYbr14jHAy2Yyau" -Content-Disposition: inline -In-Reply-To: <25849.1132238470@sss.pgh.pa.us> -X-Editor: Vim http://www.vim.org/ -X-Info: http://www.snowman.net -X-Operating-System: Linux/2.4.24ns.3.0 (i686) -X-Uptime: 10:01:46 up 159 days, 7:15, 2 users, load average: 0.31, 0.23, 0.13 -User-Agent: Mutt/1.5.9i -Status: OR - - ---aAYbr14jHAy2Yyau -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -Content-Transfer-Encoding: quoted-printable - -* Tom Lane (tgl@sss.pgh.pa.us) wrote: -> This is exactly the point --- pretty much nobody has come to us and -> asked for a feature that does what Peter and Martijn say MERGE does. -> (I haven't bothered to look at the 2003 spec, I'm assuming they read it -> correctly.) What we *have* been asked for, over and over, is an -> insert-or-update feature that's not so tedious and inefficient as the -> savepoint-insert-rollback-update kluge. That's what we ought to be -> concentrating on providing. - -I guess to be clear on what this distinction actually is, specifically: -MERGE under SQL2003 doesn't appear to be intended to be used -concurrently. For data warehousing situations this can be just fine -such as in my case where I get a monthly update of some information and -need to merge that update in with the prior information. In this case -there's only one MERGE running and I'd hope it'd be faster than doing -check for existance, insert/update on each row in plpgsql or something -(since there'd be multiple index lookups, etc, I think). Concurrent -MERGEs running *can* fail, just like whole transactions which do the -check/insert/update can fail. - -REPLACE/INSERT ON DUPLICATE UPDATE appears to essentially be a -transaction which is supposed to not fail but instead do locking to -ensure that it doesn't fail. This requires predicate locking to be -efficient because you want to tell the concurrent transaction "if you -have the same key as me, just wait a second and you can do an update -'cause I'm going to create the key if it doesn't exist before I'm done". - -I think REPLACE/INSERT ON DUPLICATE UPDATE is definitely harder to do -than MERGE because of the idea that it isn't supposed to fail generally. -I think SQL2003 MERGE would be reasonably easy to do and to get the -efficiency benefits out of it (assuming there are some to be had in the -end). =20 - -I don't think MERGE can really be made to be both though, in which case -it should really be the SQL2003 MERGE and we can make REPLACE/INSERT ON -DUPLICATE UPDATE something else. Perhaps a special form of MERGE where -you know it's going to be doing that locking. I really don't like the -idea of making the SQL2003 version of MERGE be the MERGE special case -(by requiring someone to take a table lock ahead of time or do something -else odd). - - Thanks, - - Stephen - ---aAYbr14jHAy2Yyau -Content-Type: application/pgp-signature; name="signature.asc" -Content-Description: Digital signature -Content-Disposition: inline - ------BEGIN PGP SIGNATURE----- -Version: GnuPG v1.4.2 (GNU/Linux) - -iD8DBQFDfJ6SrzgMPqB3kigRAjXWAJ9R/50PoocURxvi74g7dwhIO4akgQCcDEDG -4hGZAVR/9Age8pFtEOp4kfo= -=F91e ------END PGP SIGNATURE----- - ---aAYbr14jHAy2Yyau-- - -From pgsql-hackers-owner+M76234@postgresql.org Thu Nov 17 22:19:04 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jAI3J3O11471 - for ; Thu, 17 Nov 2005 22:19:04 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id CBDDBC4B337; - Fri, 18 Nov 2005 03:18:59 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id BB822DB600 - for ; Thu, 17 Nov 2005 23:12:08 -0400 (AST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 30987-04 - for ; - Fri, 18 Nov 2005 03:12:11 +0000 (GMT) -X-Greylist: from auto-whitelisted by SQLgrey- -Received: from candle.pha.pa.us (candle.pha.pa.us [64.139.89.126]) - by svr1.postgresql.org (Postfix) with ESMTP id 7CB16DB466 - for ; Thu, 17 Nov 2005 23:12:05 -0400 (AST) -Received: (from pgman@localhost) - by candle.pha.pa.us (8.11.6/8.11.6) id jAI3BXS10887; - Thu, 17 Nov 2005 22:11:33 -0500 (EST) -From: Bruce Momjian -Message-ID: <200511180311.jAI3BXS10887@candle.pha.pa.us> -Subject: Re: [HACKERS] MERGE vs REPLACE -In-Reply-To: <25849.1132238470@sss.pgh.pa.us> -To: Tom Lane -Date: Thu, 17 Nov 2005 22:11:33 -0500 (EST) -cc: Csaba Nagy , - Martijn van Oosterhout , - Zeugswetter Andreas DCP SD , - Dann Corbit , Simon Riggs , - Rick Gigger , - Christopher Kings-Lynne , - "Jim C. Nasby" , josh@agliodbs.com, - pgsql-hackers@postgresql.org, Jaime Casanova , - Peter Eisentraut -X-Mailer: ELM [version 2.4ME+ PL121 (25)] -MIME-Version: 1.0 -Content-Transfer-Encoding: 7bit -Content-Type: text/plain; charset=US-ASCII -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.036 required=5 tests=[AWL=0.036] -X-Spam-Score: 0.036 -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -Tom Lane wrote: -> Csaba Nagy writes: -> > OK, in this case I don't care about either MERGE or REPLACE, but for an -> > UPSERT which does the locking :-) -> -> This is exactly the point --- pretty much nobody has come to us and -> asked for a feature that does what Peter and Martijn say MERGE does. -> (I haven't bothered to look at the 2003 spec, I'm assuming they read it -> correctly.) What we *have* been asked for, over and over, is an -> insert-or-update feature that's not so tedious and inefficient as the -> savepoint-insert-rollback-update kluge. That's what we ought to be -> concentrating on providing. - -I am confused over the various options. I have heard these syntaxes: - - SQL2003 MERGE - MySQL REPLACE - http://dev.mysql.com/doc/refman/5.1/en/replace.html - MySQL INSERT VIOLATION ... - UPSERT - -So it seems MERGE does not have the use-case we most need, though it can -be bent to do it. (Given their MATCH syntax, it doesn't seem there is -any logic that it tries INSERT first). - -Looking at the MySQL URL above, REPLACE has three possible syntaxes with -normal (DELETE), SET (UPDATE), and SELECT. Is this the direction we -need to go? I don't like INSERT ... VIOLATION because I would like a -new keyword for this. Is UPSERT the same as REPLACE? Should we use -UPSERT instead? - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 359-1001 - + If your life is a hard drive, | 13 Roberts Road - + Christ can be your backup. | Newtown Square, Pennsylvania 19073 - ----------------------------(end of broadcast)--------------------------- -TIP 4: Have you searched our list archives? - - http://archives.postgresql.org - -From DCorbit@connx.com Thu Nov 17 22:43:54 2005 -Return-path: -Received: from postal.corporate.connx.com (postal.corporate.connx.com [65.212.159.187]) - by candle.pha.pa.us (8.11.6/8.11.6) with SMTP id jAI3hqO14288 - for ; Thu, 17 Nov 2005 22:43:52 -0500 (EST) -content-class: urn:content-classes:message -MIME-Version: 1.0 -Content-Type: multipart/mixed; - boundary="----_=_NextPart_001_01C5EBF2.45ACB7DC" -X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 -Subject: RE: [HACKERS] MERGE vs REPLACE -Date: Thu, 17 Nov 2005 19:43:43 -0800 -Message-ID: -Thread-Topic: [HACKERS] MERGE vs REPLACE -Thread-Index: AcXr7dUIILx3WyhrREu/dEAzLwToBgABFVZw -From: "Dann Corbit" -To: "Bruce Momjian" , "Tom Lane" -cc: "Csaba Nagy" , - "Martijn van Oosterhout" , - "Zeugswetter Andreas DCP SD" , - "Simon Riggs" , - "Rick Gigger" , - "Christopher Kings-Lynne" , - "Jim C. Nasby" , , - , "Jaime Casanova" , - "Peter Eisentraut" -Status: OR - -This is a multi-part message in MIME format. - -------_=_NextPart_001_01C5EBF2.45ACB7DC -Content-Type: text/plain; - charset="us-ascii" -Content-Transfer-Encoding: quoted-printable - -Attached web page is an extract from the SQL Standards working committee -document on MERGE - -> -----Original Message----- -> From: Bruce Momjian [mailto:pgman@candle.pha.pa.us] -> Sent: Thursday, November 17, 2005 7:12 PM -> To: Tom Lane -> Cc: Csaba Nagy; Martijn van Oosterhout; Zeugswetter Andreas DCP SD; -Dann -> Corbit; Simon Riggs; Rick Gigger; Christopher Kings-Lynne; Jim C. -Nasby; -> josh@agliodbs.com; pgsql-hackers@postgresql.org; Jaime Casanova; Peter -> Eisentraut -> Subject: Re: [HACKERS] MERGE vs REPLACE ->=20 -> Tom Lane wrote: -> > Csaba Nagy writes: -> > > OK, in this case I don't care about either MERGE or REPLACE, but -for -> an -> > > UPSERT which does the locking :-) -> > -> > This is exactly the point --- pretty much nobody has come to us and -> > asked for a feature that does what Peter and Martijn say MERGE does. -> > (I haven't bothered to look at the 2003 spec, I'm assuming they read -it -> > correctly.) What we *have* been asked for, over and over, is an -> > insert-or-update feature that's not so tedious and inefficient as -the -> > savepoint-insert-rollback-update kluge. That's what we ought to be -> > concentrating on providing. ->=20 -> I am confused over the various options. I have heard these syntaxes: ->=20 -> SQL2003 MERGE -> MySQL REPLACE -> http://dev.mysql.com/doc/refman/5.1/en/replace.html -> MySQL INSERT VIOLATION ... -> UPSERT ->=20 -> So it seems MERGE does not have the use-case we most need, though it -can -> be bent to do it. (Given their MATCH syntax, it doesn't seem there is -> any logic that it tries INSERT first). ->=20 -> Looking at the MySQL URL above, REPLACE has three possible syntaxes -with -> normal (DELETE), SET (UPDATE), and SELECT. Is this the direction we -> need to go? I don't like INSERT ... VIOLATION because I would like a -> new keyword for this. Is UPSERT the same as REPLACE? Should we use -> UPSERT instead? ->=20 -> -- -> Bruce Momjian | http://candle.pha.pa.us -> pgman@candle.pha.pa.us | (610) 359-1001 -> + If your life is a hard drive, | 13 Roberts Road -> + Christ can be your backup. | Newtown Square, Pennsylvania -> 19073 - -------_=_NextPart_001_01C5EBF2.45ACB7DC -Content-Type: text/html; - name="merge.htm" -Content-Transfer-Encoding: base64 -Content-Description: merge.htm -Content-Disposition: attachment; - filename="merge.htm" - -PGh0bWwgeG1sbnM6bz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6b2ZmaWNlIg0K -eG1sbnM6dz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCINCnhtbG5zOng9 -InVybjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOmV4Y2VsIg0KeG1sbnM9Imh0dHA6Ly93 -d3cudzMub3JnL1RSL1JFQy1odG1sNDAiPg0KDQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9Q29u -dGVudC1UeXBlIGNvbnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD13aW5kb3dzLTEyNTIiPg0KPG1l -dGEgbmFtZT1Qcm9nSWQgY29udGVudD1Xb3JkLkRvY3VtZW50Pg0KPG1ldGEgbmFtZT1HZW5lcmF0 -b3IgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTEiPg0KPG1ldGEgbmFtZT1PcmlnaW5hdG9yIGNv -bnRlbnQ9Ik1pY3Jvc29mdCBXb3JkIDExIj4NCjxsaW5rIHJlbD1GaWxlLUxpc3QgaHJlZj0ibWVy -Z2VfZmlsZXMvZmlsZWxpc3QueG1sIj4NCjx0aXRsZT4xNDwvdGl0bGU+DQo8IS0tW2lmIGd0ZSBt -c28gOV0+PHhtbD4NCiA8bzpEb2N1bWVudFByb3BlcnRpZXM+DQogIDxvOkF1dGhvcj5EYW5uIENv -cmJpdDwvbzpBdXRob3I+DQogIDxvOkxhc3RBdXRob3I+RGFubiBDb3JiaXQ8L286TGFzdEF1dGhv -cj4NCiAgPG86UmV2aXNpb24+MTwvbzpSZXZpc2lvbj4NCiAgPG86VG90YWxUaW1lPjc8L286VG90 -YWxUaW1lPg0KICA8bzpDcmVhdGVkPjIwMDUtMTEtMThUMDM6MzU6MDBaPC9vOkNyZWF0ZWQ+DQog -IDxvOkxhc3RTYXZlZD4yMDA1LTExLTE4VDAzOjQyOjAwWjwvbzpMYXN0U2F2ZWQ+DQogIDxvOlBh -Z2VzPjE8L286UGFnZXM+DQogIDxvOldvcmRzPjIzODk8L286V29yZHM+DQogIDxvOkNoYXJhY3Rl -cnM+MTM2MjE8L286Q2hhcmFjdGVycz4NCiAgPG86Q29tcGFueT5DT05OWCBTb2x1dGlvbnM8L286 -Q29tcGFueT4NCiAgPG86TGluZXM+MTEzPC9vOkxpbmVzPg0KICA8bzpQYXJhZ3JhcGhzPjMxPC9v -OlBhcmFncmFwaHM+DQogIDxvOkNoYXJhY3RlcnNXaXRoU3BhY2VzPjE1OTc5PC9vOkNoYXJhY3Rl -cnNXaXRoU3BhY2VzPg0KICA8bzpWZXJzaW9uPjExLjY1Njg8L286VmVyc2lvbj4NCiA8L286RG9j -dW1lbnRQcm9wZXJ0aWVzPg0KPC94bWw+PCFbZW5kaWZdLS0+PCEtLVtpZiBndGUgbXNvIDldPjx4 -bWw+DQogPHc6V29yZERvY3VtZW50Pg0KICA8dzpTcGVsbGluZ1N0YXRlPkNsZWFuPC93OlNwZWxs -aW5nU3RhdGU+DQogIDx3OkdyYW1tYXJTdGF0ZT5DbGVhbjwvdzpHcmFtbWFyU3RhdGU+DQogIDx3 -OlB1bmN0dWF0aW9uS2VybmluZy8+DQogIDx3OlZhbGlkYXRlQWdhaW5zdFNjaGVtYXMvPg0KICA8 -dzpTYXZlSWZYTUxJbnZhbGlkPmZhbHNlPC93OlNhdmVJZlhNTEludmFsaWQ+DQogIDx3Oklnbm9y -ZU1peGVkQ29udGVudD5mYWxzZTwvdzpJZ25vcmVNaXhlZENvbnRlbnQ+DQogIDx3OkFsd2F5c1No -b3dQbGFjZWhvbGRlclRleHQ+ZmFsc2U8L3c6QWx3YXlzU2hvd1BsYWNlaG9sZGVyVGV4dD4NCiAg -PHc6Q29tcGF0aWJpbGl0eT4NCiAgIDx3OkJyZWFrV3JhcHBlZFRhYmxlcy8+DQogICA8dzpTbmFw -VG9HcmlkSW5DZWxsLz4NCiAgIDx3OldyYXBUZXh0V2l0aFB1bmN0Lz4NCiAgIDx3OlVzZUFzaWFu -QnJlYWtSdWxlcy8+DQogICA8dzpEb250R3Jvd0F1dG9maXQvPg0KICA8L3c6Q29tcGF0aWJpbGl0 -eT4NCiAgPHc6QnJvd3NlckxldmVsPk1pY3Jvc29mdEludGVybmV0RXhwbG9yZXI0PC93OkJyb3dz -ZXJMZXZlbD4NCiA8L3c6V29yZERvY3VtZW50Pg0KPC94bWw+PCFbZW5kaWZdLS0+PCEtLVtpZiBn -dGUgbXNvIDldPjx4bWw+DQogPHc6TGF0ZW50U3R5bGVzIERlZkxvY2tlZFN0YXRlPSJmYWxzZSIg -TGF0ZW50U3R5bGVDb3VudD0iMTU2Ij4NCiA8L3c6TGF0ZW50U3R5bGVzPg0KPC94bWw+PCFbZW5k -aWZdLS0+DQo8c3R5bGU+DQo8IS0tDQogLyogRm9udCBEZWZpbml0aW9ucyAqLw0KIEBmb250LWZh -Y2UNCgl7Zm9udC1mYW1pbHk6Q291cmllcjsNCglwYW5vc2UtMToyIDcgNCA5IDIgMiA1IDIgNCA0 -Ow0KCW1zby1mb250LWNoYXJzZXQ6MDsNCgltc28tZ2VuZXJpYy1mb250LWZhbWlseTptb2Rlcm47 -DQoJbXNvLWZvbnQtZm9ybWF0Om90aGVyOw0KCW1zby1mb250LXBpdGNoOmZpeGVkOw0KCW1zby1m -b250LXNpZ25hdHVyZTozIDAgMCAwIDEgMDt9DQogLyogU3R5bGUgRGVmaW5pdGlvbnMgKi8NCiBw -Lk1zb05vcm1hbCwgbGkuTXNvTm9ybWFsLCBkaXYuTXNvTm9ybWFsDQoJe21zby1zdHlsZS1wYXJl -bnQ6IiI7DQoJbWFyZ2luOjBpbjsNCgltYXJnaW4tYm90dG9tOi4wMDAxcHQ7DQoJbXNvLXBhZ2lu -YXRpb246d2lkb3ctb3JwaGFuOw0KCWZvbnQtc2l6ZToxMi4wcHQ7DQoJZm9udC1mYW1pbHk6IlRp -bWVzIE5ldyBSb21hbiI7DQoJbXNvLWZhcmVhc3QtZm9udC1mYW1pbHk6IlRpbWVzIE5ldyBSb21h -biI7fQ0Kc3Bhbi5TcGVsbEUNCgl7bXNvLXN0eWxlLW5hbWU6IiI7DQoJbXNvLXNwbC1lOnllczt9 -DQpzcGFuLkdyYW1FDQoJe21zby1zdHlsZS1uYW1lOiIiOw0KCW1zby1ncmFtLWU6eWVzO30NCkBw -YWdlIFNlY3Rpb24xDQoJe3NpemU6OC41aW4gMTEuMGluOw0KCW1hcmdpbjouNzVpbiAuNzVpbiAu -NzVpbiAuNzVpbjsNCgltc28taGVhZGVyLW1hcmdpbjouNWluOw0KCW1zby1mb290ZXItbWFyZ2lu -Oi41aW47DQoJbXNvLXBhcGVyLXNvdXJjZTowO30NCmRpdi5TZWN0aW9uMQ0KCXtwYWdlOlNlY3Rp -b24xO30NCi0tPg0KPC9zdHlsZT4NCjwhLS1baWYgZ3RlIG1zbyAxMF0+DQo8c3R5bGU+DQogLyog -U3R5bGUgRGVmaW5pdGlvbnMgKi8NCiB0YWJsZS5Nc29Ob3JtYWxUYWJsZQ0KCXttc28tc3R5bGUt -bmFtZToiVGFibGUgTm9ybWFsIjsNCgltc28tdHN0eWxlLXJvd2JhbmQtc2l6ZTowOw0KCW1zby10 -c3R5bGUtY29sYmFuZC1zaXplOjA7DQoJbXNvLXN0eWxlLW5vc2hvdzp5ZXM7DQoJbXNvLXN0eWxl -LXBhcmVudDoiIjsNCgltc28tcGFkZGluZy1hbHQ6MGluIDUuNHB0IDBpbiA1LjRwdDsNCgltc28t -cGFyYS1tYXJnaW46MGluOw0KCW1zby1wYXJhLW1hcmdpbi1ib3R0b206LjAwMDFwdDsNCgltc28t -cGFnaW5hdGlvbjp3aWRvdy1vcnBoYW47DQoJZm9udC1zaXplOjEwLjBwdDsNCglmb250LWZhbWls -eToiVGltZXMgTmV3IFJvbWFuIjsNCgltc28tYW5zaS1sYW5ndWFnZTojMDQwMDsNCgltc28tZmFy -ZWFzdC1sYW5ndWFnZTojMDQwMDsNCgltc28tYmlkaS1sYW5ndWFnZTojMDQwMDt9DQo8L3N0eWxl -Pg0KPCFbZW5kaWZdLS0+DQo8L2hlYWQ+DQoNCjxib2R5IGxhbmc9RU4tVVMgc3R5bGU9J3RhYi1p -bnRlcnZhbDouNWluJz4NCg0KPGRpdiBjbGFzcz1TZWN0aW9uMT4NCg0KPHAgY2xhc3M9TXNvTm9y -bWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25l -Jz48Yj48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxNC4wcHQ7Y29sb3I6YmxhY2snPjE0LjkgJmx0 -O21lcmdlIHN0YXRlbWVudCZndDs8bzpwPjwvbzpwPjwvc3Bhbj48L2I+PC9wPg0KDQo8cCBjbGFz -cz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3Nw -YWNlOm5vbmUnPjxiPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjE0LjBwdDtjb2xvcjpibGFjayc+ -RnVuY3Rpb248bzpwPjwvbzpwPjwvc3Bhbj48L2I+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwg -c3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxz -cGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+Q29uZGl0aW9uYWxseSB1 -cGRhdGUgcm93cyBvZiBhIHRhYmxlLCBvcg0KaW5zZXJ0IG5ldyByb3dzIGludG8gYSB0YWJsZSwg -b3IgYm90aC48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHls -ZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PGI+PHNw -YW4NCnN0eWxlPSdmb250LXNpemU6MTQuMHB0O2NvbG9yOmJsYWNrJz5Gb3JtYXQ8bzpwPjwvbzpw -Pjwvc3Bhbj48L2I+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQt -Z3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1z -aXplOjkuMHB0O2ZvbnQtZmFtaWx5OkNvdXJpZXI7bXNvLWJpZGktZm9udC1mYW1pbHk6Q291cmll -cjsNCmNvbG9yOmJsYWNrJz4mbHQ7PHNwYW4gY2xhc3M9R3JhbUU+bWVyZ2U8L3NwYW4+IHN0YXRl -bWVudCZndDsgOjo9PG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwg -c3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxz -cGFuDQpzdHlsZT0nZm9udC1zaXplOjkuMHB0O2ZvbnQtZmFtaWx5OkNvdXJpZXI7bXNvLWJpZGkt -Zm9udC1mYW1pbHk6Q291cmllcjsNCmNvbG9yOmJsYWNrJz5NRVJHRSBJTlRPICZsdDt0YXJnZXQg -dGFibGUmZ3Q7IDxzcGFuIGNsYXNzPUdyYW1FPlsgWzwvc3Bhbj4gQVMgXQ0KJmx0O21lcmdlIGNv -cnJlbGF0aW9uIG5hbWUmZ3Q7IF08bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1z -b05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6 -bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6OS4wcHQ7Zm9udC1mYW1pbHk6Q291cmllcjtt -c28tYmlkaS1mb250LWZhbWlseTpDb3VyaWVyOw0KY29sb3I6YmxhY2snPlVTSU5HICZsdDt0YWJs -ZSByZWZlcmVuY2UmZ3Q7PG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3Jt -YWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUn -PjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjkuMHB0O2ZvbnQtZmFtaWx5OkNvdXJpZXI7bXNvLWJp -ZGktZm9udC1mYW1pbHk6Q291cmllcjsNCmNvbG9yOmJsYWNrJz5PTiAmbHQ7c2VhcmNoIGNvbmRp -dGlvbiZndDsgJmx0O21lcmdlIG9wZXJhdGlvbiBzcGVjaWZpY2F0aW9uJmd0OzxvOnA+PC9vOnA+ -PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQt -YWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZTo5 -LjBwdDtmb250LWZhbWlseTpDb3VyaWVyO21zby1iaWRpLWZvbnQtZmFtaWx5OkNvdXJpZXI7DQpj -b2xvcjpibGFjayc+Jmx0OzxzcGFuIGNsYXNzPUdyYW1FPm1lcmdlPC9zcGFuPiBjb3JyZWxhdGlv -biBuYW1lJmd0OyA6Oj0NCiZsdDtjb3JyZWxhdGlvbiBuYW1lJmd0OzxvOnA+PC9vOnA+PC9zcGFu -PjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246 -bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZTo5LjBwdDtm -b250LWZhbWlseTpDb3VyaWVyO21zby1iaWRpLWZvbnQtZmFtaWx5OkNvdXJpZXI7DQpjb2xvcjpi -bGFjayc+Jmx0OzxzcGFuIGNsYXNzPUdyYW1FPm1lcmdlPC9zcGFuPiBvcGVyYXRpb24gc3BlY2lm -aWNhdGlvbiZndDsgOjo9DQombHQ7bWVyZ2Ugd2hlbiBjbGF1c2UmZ3Q7Li4uPG86cD48L286cD48 -L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1h -bGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjku -MHB0O2ZvbnQtZmFtaWx5OkNvdXJpZXI7bXNvLWJpZGktZm9udC1mYW1pbHk6Q291cmllcjsNCmNv -bG9yOmJsYWNrJz4mbHQ7bWVyZ2Ugd2hlbiBjbGF1c2U8c3BhbiBjbGFzcz1HcmFtRT4mZ3Q7IDo8 -L3NwYW4+Oj08bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHls -ZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4N -CnN0eWxlPSdmb250LXNpemU6OS4wcHQ7Zm9udC1mYW1pbHk6Q291cmllcjttc28tYmlkaS1mb250 -LWZhbWlseTpDb3VyaWVyOw0KY29sb3I6YmxhY2snPiZsdDttZXJnZSB3aGVuIG1hdGNoZWQgY2xh -dXNlJmd0OzxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxl -PSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0K -c3R5bGU9J2ZvbnQtc2l6ZTo5LjBwdDtmb250LWZhbWlseTpDb3VyaWVyO21zby1iaWRpLWZvbnQt -ZmFtaWx5OkNvdXJpZXI7DQpjb2xvcjpibGFjayc+fCAmbHQ7bWVyZ2Ugd2hlbiBub3QgbWF0Y2hl -ZCBjbGF1c2UmZ3Q7PG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwg -c3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxz -cGFuDQpzdHlsZT0nZm9udC1zaXplOjkuMHB0O2ZvbnQtZmFtaWx5OkNvdXJpZXI7bXNvLWJpZGkt -Zm9udC1mYW1pbHk6Q291cmllcjsNCmNvbG9yOmJsYWNrJz4mbHQ7bWVyZ2Ugd2hlbiBtYXRjaGVk -IGNsYXVzZTxzcGFuIGNsYXNzPUdyYW1FPiZndDsgOjwvc3Bhbj46PTxvOnA+PC9vOnA+PC9zcGFu -PjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246 -bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZTo5LjBwdDtm -b250LWZhbWlseTpDb3VyaWVyO21zby1iaWRpLWZvbnQtZmFtaWx5OkNvdXJpZXI7DQpjb2xvcjpi -bGFjayc+V0hFTiBNQVRDSEVEIFRIRU4gJmx0O21lcmdlIHVwZGF0ZSBzcGVjaWZpY2F0aW9uJmd0 -OzxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28t -bGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9 -J2ZvbnQtc2l6ZTo5LjBwdDtmb250LWZhbWlseTpDb3VyaWVyO21zby1iaWRpLWZvbnQtZmFtaWx5 -OkNvdXJpZXI7DQpjb2xvcjpibGFjayc+Jmx0OzxzcGFuIGNsYXNzPUdyYW1FPm1lcmdlPC9zcGFu -PiB3aGVuIG5vdCBtYXRjaGVkIGNsYXVzZSZndDsgOjo9PG86cD48L286cD48L3NwYW4+PC9wPg0K -DQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3Rl -eHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjkuMHB0O2ZvbnQtZmFt -aWx5OkNvdXJpZXI7bXNvLWJpZGktZm9udC1mYW1pbHk6Q291cmllcjsNCmNvbG9yOmJsYWNrJz5X -SEVOIE5PVCBNQVRDSEVEIFRIRU4gJmx0O21lcmdlIGluc2VydCBzcGVjaWZpY2F0aW9uJmd0Ozxv -OnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5 -b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2Zv -bnQtc2l6ZTo5LjBwdDtmb250LWZhbWlseTpDb3VyaWVyO21zby1iaWRpLWZvbnQtZmFtaWx5OkNv -dXJpZXI7DQpjb2xvcjpibGFjayc+Jmx0OzxzcGFuIGNsYXNzPUdyYW1FPm1lcmdlPC9zcGFuPiB1 -cGRhdGUgc3BlY2lmaWNhdGlvbiZndDsgOjo9DQpVUERBVEUgU0VUICZsdDtzZXQgY2xhdXNlIGxp -c3QmZ3Q7PG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9 -J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpz -dHlsZT0nZm9udC1zaXplOjkuMHB0O2ZvbnQtZmFtaWx5OkNvdXJpZXI7bXNvLWJpZGktZm9udC1m -YW1pbHk6Q291cmllcjsNCmNvbG9yOmJsYWNrJz4mbHQ7PHNwYW4gY2xhc3M9R3JhbUU+bWVyZ2U8 -L3NwYW4+IGluc2VydCBzcGVjaWZpY2F0aW9uJmd0OyA6Oj08bzpwPjwvbzpwPjwvc3Bhbj48L3A+ -DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7 -dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6OS4wcHQ7Zm9udC1m -YW1pbHk6Q291cmllcjttc28tYmlkaS1mb250LWZhbWlseTpDb3VyaWVyOw0KY29sb3I6YmxhY2sn -PklOU0VSVCA8c3BhbiBjbGFzcz1HcmFtRT5bICZsdDs8L3NwYW4+bGVmdCA8c3BhbiBjbGFzcz1T -cGVsbEU+cGFyZW48L3NwYW4+Jmd0Ow0KJmx0O2luc2VydCBjb2x1bW4gbGlzdCZndDsgJmx0O3Jp -Z2h0IDxzcGFuIGNsYXNzPVNwZWxsRT5wYXJlbjwvc3Bhbj4mZ3Q7IF08bzpwPjwvbzpwPjwvc3Bh -bj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWdu -Om5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCmNsYXNzPUdyYW1FPjxzcGFuIHN0eWxl -PSdmb250LXNpemU6OS4wcHQ7Zm9udC1mYW1pbHk6Q291cmllcjttc28tYmlkaS1mb250LWZhbWls -eToNCkNvdXJpZXI7Y29sb3I6YmxhY2snPlsgJmx0Ozwvc3Bhbj48L3NwYW4+PHNwYW4gc3R5bGU9 -J2ZvbnQtc2l6ZTo5LjBwdDsNCmZvbnQtZmFtaWx5OkNvdXJpZXI7bXNvLWJpZGktZm9udC1mYW1p -bHk6Q291cmllcjtjb2xvcjpibGFjayc+b3ZlcnJpZGUNCmNsYXVzZSZndDsgXTxvOnA+PC9vOnA+ -PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQt -YWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZTo5 -LjBwdDtmb250LWZhbWlseTpDb3VyaWVyO21zby1iaWRpLWZvbnQtZmFtaWx5OkNvdXJpZXI7DQpj -b2xvcjpibGFjayc+VkFMVUVTICZsdDttZXJnZSBpbnNlcnQgdmFsdWUgbGlzdCZndDs8bzpwPjwv -bzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1n -cmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNp -emU6OS4wcHQ7Zm9udC1mYW1pbHk6Q291cmllcjttc28tYmlkaS1mb250LWZhbWlseTpDb3VyaWVy -Ow0KY29sb3I6YmxhY2snPiZsdDs8c3BhbiBjbGFzcz1HcmFtRT5tZXJnZTwvc3Bhbj4gaW5zZXJ0 -IHZhbHVlIGxpc3QmZ3Q7IDo6PTxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNv -Tm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpu -b25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZTo5LjBwdDtmb250LWZhbWlseTpDb3VyaWVyO21z -by1iaWRpLWZvbnQtZmFtaWx5OkNvdXJpZXI7DQpjb2xvcjpibGFjayc+Jmx0OzxzcGFuIGNsYXNz -PUdyYW1FPmxlZnQ8L3NwYW4+IDxzcGFuIGNsYXNzPVNwZWxsRT5wYXJlbjwvc3Bhbj4mZ3Q7PG86 -cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlv -dXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9u -dC1zaXplOjkuMHB0O2ZvbnQtZmFtaWx5OkNvdXJpZXI7bXNvLWJpZGktZm9udC1mYW1pbHk6Q291 -cmllcjsNCmNvbG9yOmJsYWNrJz4mbHQ7PHNwYW4gY2xhc3M9R3JhbUU+bWVyZ2U8L3NwYW4+IGlu -c2VydCB2YWx1ZSBlbGVtZW50Jmd0OyBbIHsNCiZsdDtjb21tYSZndDsgJmx0O21lcmdlIGluc2Vy -dCB2YWx1ZSBlbGVtZW50Jmd0OyB9Li4uIF08bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNs -YXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRv -c3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6OS4wcHQ7Zm9udC1mYW1pbHk6Q291 -cmllcjttc28tYmlkaS1mb250LWZhbWlseTpDb3VyaWVyOw0KY29sb3I6YmxhY2snPiZsdDs8c3Bh -biBjbGFzcz1HcmFtRT5yaWdodDwvc3Bhbj4gPHNwYW4gY2xhc3M9U3BlbGxFPnBhcmVuPC9zcGFu -PiZndDs8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0n -bXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0 -eWxlPSdmb250LXNpemU6OS4wcHQ7Zm9udC1mYW1pbHk6Q291cmllcjttc28tYmlkaS1mb250LWZh -bWlseTpDb3VyaWVyOw0KY29sb3I6YmxhY2snPiZsdDs8c3BhbiBjbGFzcz1HcmFtRT5tZXJnZTwv -c3Bhbj4gaW5zZXJ0IHZhbHVlIGVsZW1lbnQmZ3Q7IDo6PTxvOnA+PC9vOnA+PC9zcGFuPjwvcD4N -Cg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0 -ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZTo5LjBwdDtmb250LWZh -bWlseTpDb3VyaWVyO21zby1iaWRpLWZvbnQtZmFtaWx5OkNvdXJpZXI7DQpjb2xvcjpibGFjayc+ -Jmx0OzxzcGFuIGNsYXNzPUdyYW1FPnZhbHVlPC9zcGFuPiBleHByZXNzaW9uJmd0OzxvOnA+PC9v -OnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdy -aWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6 -ZTo5LjBwdDtmb250LWZhbWlseTpDb3VyaWVyO21zby1iaWRpLWZvbnQtZmFtaWx5OkNvdXJpZXI7 -DQpjb2xvcjpibGFjayc+fCAmbHQ7Y29udGV4dHVhbGx5IHR5cGVkIHZhbHVlIHNwZWNpZmljYXRp -b24mZ3Q7PG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9 -J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxiPjxzcGFu -DQpzdHlsZT0nZm9udC1zaXplOjE0LjBwdDtjb2xvcjpibGFjayc+U3ludGF4IFJ1bGVzPG86cD48 -L286cD48L3NwYW4+PC9iPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5 -b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2Zv -bnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPjEpIE5laXRoZXIgJmx0O21lcmdlIHdoZW4gbWF0 -Y2hlZA0KY2xhdXNlJmd0OyBub3IgJmx0O21lcmdlIHdoZW4gbm90IG1hdGNoZWQgY2xhdXNlJmd0 -OyBzaGFsbCBiZSBzcGVjaWZpZWQgbW9yZQ0KdGhhbiBvbmNlLjxvOnA+PC9vOnA+PC9zcGFuPjwv -cD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9u -ZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29s -b3I6YmxhY2snPjIpIExldCA8aT5UTiA8L2k+YmUgdGhlICZsdDt0YWJsZSBuYW1lJmd0Ow0KY29u -dGFpbmVkIGluICZsdDt0YXJnZXQgdGFibGUmZ3Q7IGFuZCBsZXQgPGk+VCA8L2k+YmUgdGhlIHRh -YmxlIGlkZW50aWZpZWQgYnkgPGk+VE48L2k+Lg0KPGk+VCA8L2k+aXMgdGhlIDxpPnN1YmplY3Qg -dGFibGUgPC9pPm9mIHRoZSAmbHQ7bWVyZ2Ugc3RhdGVtZW50Jmd0Oy48bzpwPjwvbzpwPjwvc3Bh -bj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWdu -Om5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0 -O2NvbG9yOmJsYWNrJz4zKSA8aT5UIDwvaT5zaGFsbCBiZSA8c3BhbiBjbGFzcz1TcGVsbEU+aW5z -ZXJ0YWJsZTwvc3Bhbj4taW50by48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1z -b05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6 -bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz40KSA8aT5U -IDwvaT5zaGFsbCBub3QgYmUgYW4gb2xkIHRyYW5zaXRpb24NCnRhYmxlIG9yIGEgbmV3IHRyYW5z -aXRpb24gdGFibGUuPG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwg -c3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxi -PjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjEwLjBwdDtjb2xvcjpibGFjayc+RGF0YSBtYW5pcHVs -YXRpb24gODA1PG86cD48L286cD48L3NwYW4+PC9iPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFs -IHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48 -Yj48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMC4wcHQ7Y29sb3I6YmxhY2snPldEIElTTy9JRUMg -OTA3NS0yOjIwMDcgKEUpPG86cD48L286cD48L3NwYW4+PC9iPjwvcD4NCg0KPHAgY2xhc3M9TXNv -Tm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpu -b25lJz48Yj48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMC4wcHQ7Y29sb3I6YmxhY2snPjE0Ljkg -Jmx0O21lcmdlIHN0YXRlbWVudCZndDs8bzpwPjwvbzpwPjwvc3Bhbj48L2I+PC9wPg0KDQo8cCBj -bGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0 -b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+ -NSkgRm9yIGVhY2ggbGVhZiBnZW5lcmFsbHkgdW5kZXJseWluZw0KdGFibGUgb2YgPGk+VCA8L2k+ -d2hvc2UgZGVzY3JpcHRvciBpbmNsdWRlcyBhIHVzZXItZGVmaW5lZCB0eXBlIG5hbWUgPGk+VURU -TjwvaT4sDQp0aGUgZGF0YSB0eXBlIGRlc2NyaXB0b3Igb2YgdGhlIHVzZXItZGVmaW5lZCB0eXBl -IDxpPlVEVCA8L2k+aWRlbnRpZmllZCBieSA8aT5VRFRODQo8L2k+c2hhbGwgaW5kaWNhdGUgdGhh -dCA8aT5VRFQgPC9pPmlzIDxzcGFuIGNsYXNzPVNwZWxsRT5pbnN0YW50aWFibGU8L3NwYW4+Ljxv -OnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5 -b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2Zv -bnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPjYpIElmIDxpPlQgPC9pPmlzIGEgdmlldywgdGhl -biAmbHQ7dGFyZ2V0DQp0YWJsZSZndDsgaXMgZWZmZWN0aXZlbHkgcmVwbGFjZWQgYnk6PG86cD48 -L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQt -Z3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1z -aXplOjkuMHB0O2ZvbnQtZmFtaWx5OkNvdXJpZXI7bXNvLWJpZGktZm9udC1mYW1pbHk6Q291cmll -cjsNCmNvbG9yOmJsYWNrJz5PTkxZIDxzcGFuIGNsYXNzPUdyYW1FPiggPGk+VE48L2k+PC9zcGFu -PjxpPiA8L2k+KTxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0 -eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bh -bg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPjcpIENhc2U6PG86cD48L286 -cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3Jp -ZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXpl -OjExLjBwdDtjb2xvcjpibGFjayc+YSkgSWYgJmx0O21lcmdlIGNvcnJlbGF0aW9uIG5hbWUmZ3Q7 -IGlzDQpzcGVjaWZpZWQsIHRoZW4gbGV0IDxpPkNOIDwvaT5iZSB0aGUgJmx0O2NvcnJlbGF0aW9u -IG5hbWUmZ3Q7IGNvbnRhaW5lZCBpbg0KJmx0O21lcmdlIGNvcnJlbGF0aW9uIG5hbWUmZ3Q7Ljxv -OnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5 -b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2Zv -bnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPmIpIE90aGVyd2lzZSwgbGV0IDxpPkNOIDwvaT5i -ZSB0aGUNCiZsdDt0YWJsZSBuYW1lJmd0OyBjb250YWluZWQgaW4gJmx0O3RhcmdldCB0YWJsZSZn -dDsuPG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21z -by1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHls -ZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+OCkgVGhlIHNjb3BlIG9mIDxpPkNOIDwv -aT5pcyAmbHQ7c2VhcmNoDQpjb25kaXRpb24mZ3Q7IGFuZCAmbHQ7c2V0IGNsYXVzZSBsaXN0Jmd0 -Oy48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNv -LWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxl -PSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz45KSBMZXQgPGk+VFIgPC9pPmJlIHRoZSAm -bHQ7dGFibGUNCnJlZmVyZW5jZSZndDsgaW1tZWRpYXRlbHkgY29udGFpbmVkIGluICZsdDttZXJn -ZSBzdGF0ZW1lbnQmZ3Q7LiA8aT5UUiA8L2k+c2hhbGwNCm5vdCBkaXJlY3RseSBjb250YWluIGEg -Jmx0O2pvaW5lZCB0YWJsZSZndDsuPG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1N -c29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNl -Om5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+MTApIFRo -ZSAmbHQ7Y29ycmVsYXRpb24gbmFtZSZndDsgb3INCmV4cG9zZWQgJmx0O3RhYmxlIG5hbWUmZ3Q7 -IHRoYXQgaXMgZXhwb3NlZCBieSA8aT5UUiA8L2k+c2hhbGwgbm90IGJlIGVxdWl2YWxlbnQNCnRv -IDxpPkNOPC9pPi48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBz -dHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNw -YW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz4xMSkgSWYgdGhlICZsdDtp -bnNlcnQgY29sdW1uIGxpc3QmZ3Q7IGlzDQpvbWl0dGVkLCB0aGVuIGFuICZsdDtpbnNlcnQgY29s -dW1uIGxpc3QmZ3Q7IHRoYXQgaWRlbnRpZmllcyBhbGwgY29sdW1ucyBvZiA8aT5UDQo8L2k+aW4g -dGhlIGFzY2VuZGluZyBzZXF1ZW5jZSBvZiB0aGVpciBvcmRpbmFsIHBvc2l0aW9uIHdpdGhpbiA8 -aT5UIDwvaT5pcw0KaW1wbGljaXQuPG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1N -c29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNl -Om5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+MTIpIENh -c2U6PG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21z -by1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHls -ZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+YSkgSWYgPGk+VCA8L2k+aXMgYSA8c3Bh -biBjbGFzcz1TcGVsbEU+cmVmZXJlbmNlYWJsZTwvc3Bhbj4NCnRhYmxlIG9yIGEgdGFibGUgaGF2 -aW5nIGFuIGlkZW50aXR5IGNvbHVtbiB3aG9zZSBkZXNjcmlwdG9yIGluY2x1ZGVzIGFuDQppbmRp -Y2F0aW9uIHRoYXQgdmFsdWVzIGFyZSBhbHdheXMgZ2VuZXJhdGVkLCB0aGVuOjxvOnA+PC9vOnA+ -PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQt -YWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0KY2xhc3M9U3BlbGxFPjxzcGFu -IHN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz5pPC9zcGFuPjwvc3Bhbj48c3Bh -bg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPikgTGV0IDxpPkMgPC9pPmJl -IHRoZSBzZWxmLXJlZmVyZW5jaW5nDQpjb2x1bW4gb3IgaWRlbnRpdHkgY29sdW1uIG9mIDxpPlQ8 -L2k+LjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdt -c28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5 -bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPmlpKSBJZiA8aT5DIDwvaT5pcyBhbiBp -ZGVudGl0eSBjb2x1bW4sIGENCnN5c3RlbS1nZW5lcmF0ZWQgc2VsZi1yZWZlcmVuY2luZyBjb2x1 -bW4gb3IgYSBkZXJpdmVkIHNlbGYtcmVmZXJlbmNpbmcgY29sdW1uDQphbmQgPGk+QyA8L2k+aXMg -Y29udGFpbmVkIGluICZsdDtpbnNlcnQgY29sdW1uIGxpc3QmZ3Q7LCB0aGVuICZsdDtvdmVycmlk -ZQ0KY2xhdXNlJmd0OyBzaGFsbCBiZSBzcGVjaWZpZWQ7IG90aGVyd2lzZSwgJmx0O292ZXJyaWRl -IGNsYXVzZSZndDsgc2hhbGwgbm90IGJlDQpzcGVjaWZpZWQuPG86cD48L286cD48L3NwYW4+PC9w -Pg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25l -O3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpjbGFzcz1HcmFtRT48c3BhbiBzdHlsZT0nZm9u -dC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+Yjwvc3Bhbj48L3NwYW4+PHNwYW4NCnN0eWxlPSdm -b250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz4pIE90aGVyd2lzZSwgJmx0O292ZXJyaWRlIGNs -YXVzZSZndDsgc2hhbGwNCm5vdCBiZSBzcGVjaWZpZWQuPG86cD48L286cD48L3NwYW4+PC9wPg0K -DQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3Rl -eHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpi -bGFjayc+MTMpIFRoZSAmbHQ7c2VhcmNoIGNvbmRpdGlvbiZndDsgc2hhbGwgbm90DQpnZW5lcmFs -bHkgY29udGFpbiBhICZsdDtyb3V0aW5lIGludm9jYXRpb24mZ3Q7IHdob3NlIHN1YmplY3Qgcm91 -dGluZSBpcyBhIDxzcGFuDQpjbGFzcz1TcGVsbEU+U1FMaW52b2tlZDwvc3Bhbj4gcm91dGluZSB0 -aGF0IHBvc3NpYmx5IG1vZGlmaWVzIFNRTC1kYXRhLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0K -PHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0 -LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6Ymxh -Y2snPjE0KSBFYWNoIGNvbHVtbiBpZGVudGlmaWVkIGJ5IGFuDQombHQ7b2JqZWN0IGNvbHVtbiZn -dDsgaW4gdGhlICZsdDtzZXQgY2xhdXNlIGxpc3QmZ3Q7IGlzIGFuIDxpPnVwZGF0ZSBvYmplY3QN -CmNvbHVtbjwvaT4uIEVhY2ggY29sdW1uIGlkZW50aWZpZWQgYnkgYSAmbHQ7Y29sdW1uIG5hbWUm -Z3Q7IGluIHRoZSBpbXBsaWNpdCBvcg0KZXhwbGljaXQgJmx0O2luc2VydCBjb2x1bW4gbGlzdCZn -dDsgaXMgYW4gPGk+aW5zZXJ0IG9iamVjdCBjb2x1bW48L2k+LiBFYWNoDQp1cGRhdGUgb2JqZWN0 -IGNvbHVtbiBhbmQgZWFjaCBpbnNlcnQgb2JqZWN0IGNvbHVtbiBpcyBhbiA8aT5vYmplY3QgY29s -dW1uPC9pPi48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHls -ZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4N -CnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz4xNSkgRXZlcnkgb2JqZWN0IGNv -bHVtbiBzaGFsbCBpZGVudGlmeSBhbg0KdXBkYXRhYmxlIGNvbHVtbiBvZiA8aT5UPC9pPi48bzpw -PjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91 -dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250 -LXNpemU6OS4wcHQ7Y29sb3I6YmxhY2snPk5PVEUgMzc5IJcgPHNwYW4gY2xhc3M9R3JhbUU+VGhl -PC9zcGFuPg0Kbm90aW9uIG9mIHVwZGF0YWJsZSBjb2x1bW5zIG9mIGJhc2UgdGFibGVzIGlzIGRl -ZmluZWQgaW4gPC9zcGFuPjxzcGFuDQpjbGFzcz1TcGVsbEU+PHNwYW4gc3R5bGU9J2ZvbnQtc2l6 -ZTo5LjBwdDtjb2xvcjojMDAwMDcwJz5TdWJjbGF1c2U8L3NwYW4+PC9zcGFuPjxzcGFuDQpzdHls -ZT0nZm9udC1zaXplOjkuMHB0O2NvbG9yOiMwMDAwNzAnPiA0LjE0LCCTVGFibGVzlDwvc3Bhbj48 -c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZTo5LjBwdDtjb2xvcjpibGFjayc+LiBUaGUgbm90aW9uIG9m -IHVwZGF0YWJsZSBjb2x1bW5zIG9mIHZpZXdlZA0KdGFibGVzIGlzIGRlZmluZWQgaW4gPC9zcGFu -PjxzcGFuIGNsYXNzPVNwZWxsRT48c3BhbiBzdHlsZT0nZm9udC1zaXplOjkuMHB0Ow0KY29sb3I6 -IzAwMDA3MCc+U3ViY2xhdXNlPC9zcGFuPjwvc3Bhbj48c3BhbiBzdHlsZT0nZm9udC1zaXplOjku -MHB0O2NvbG9yOiMwMDAwNzAnPg0KMTEuMjIsIJMmbHQ7dmlldyBkZWZpbml0aW9uJmd0O5Q8L3Nw -YW4+PHNwYW4gc3R5bGU9J2ZvbnQtc2l6ZTo5LjBwdDtjb2xvcjpibGFjayc+LjxvOnA+PC9vOnA+ -PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQt -YWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZTox -MS4wcHQ7Y29sb3I6YmxhY2snPjE2KSBObyAmbHQ7Y29sdW1uIG5hbWUmZ3Q7IG9mIDxpPlQgPC9p -PnNoYWxsDQpiZSBpZGVudGlmaWVkIG1vcmUgdGhhbiBvbmNlIGluIDxzcGFuIGNsYXNzPVNwZWxs -RT5pbjwvc3Bhbj4gYW4gJmx0O2luc2VydA0KY29sdW1uIGxpc3QmZ3Q7LjxvOnA+PC9vOnA+PC9z -cGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxp -Z246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4w -cHQ7Y29sb3I6YmxhY2snPjE3KSBMZXQgPGk+TkkgPC9pPmJlIHRoZSBudW1iZXIgb2YNCiZsdDtt -ZXJnZSBpbnNlcnQgdmFsdWUgZWxlbWVudCZndDtzIGNvbnRhaW5lZCBpbiAmbHQ7bWVyZ2UgaW5z -ZXJ0IHZhbHVlDQpsaXN0Jmd0Oy4gTGV0IDxpPkVYUDwvaT48L3NwYW4+PHNwYW4gc3R5bGU9J2Zv -bnQtc2l6ZTo5LjBwdDtjb2xvcjpibGFjayc+MTwvc3Bhbj48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6 -ZToxMS4wcHQ7Y29sb3I6YmxhY2snPiwgPGk+RVhQPC9pPjwvc3Bhbj48c3BhbiBzdHlsZT0nZm9u -dC1zaXplOg0KOS4wcHQ7Y29sb3I6YmxhY2snPjI8L3NwYW4+PHNwYW4gc3R5bGU9J2ZvbnQtc2l6 -ZToxMS4wcHQ7Y29sb3I6YmxhY2snPiwgPHNwYW4NCmNsYXNzPUdyYW1FPi4uLiAsPC9zcGFuPiA8 -aT5FWFA8L2k+PC9zcGFuPjxpPjxzcGFuIHN0eWxlPSdmb250LXNpemU6OS4wcHQ7DQpjb2xvcjpi -bGFjayc+TkkgPC9zcGFuPjwvaT48c3BhbiBzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpi -bGFjayc+YmUgdGhvc2UNCiZsdDttZXJnZSBpbnNlcnQgdmFsdWUgZWxlbWVudCZndDtzLjxvOnA+ -PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0 -LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQt -c2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPjE4KSBUaGUgbnVtYmVyIG9mICZsdDtjb2x1bW4gbmFt -ZSZndDtzIGluDQp0aGUgJmx0O2luc2VydCBjb2x1bW4gbGlzdCZndDsgc2hhbGwgYmUgZXF1YWwg -dG8gPGk+Tkk8L2k+LjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFs -IHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48 -c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPjE5KSBUaGUgZGVjbGFy -ZWQgdHlwZSBvZiBldmVyeQ0KJmx0O2NvbnRleHR1YWxseSB0eXBlZCB2YWx1ZSBzcGVjaWZpY2F0 -aW9uJmd0OyA8aT5DVlMgPC9pPmluIGEgJmx0O21lcmdlIGluc2VydA0KdmFsdWUgbGlzdCZndDsg -aXMgdGhlIGRhdGEgdHlwZSA8aT5EVCA8L2k+aW5kaWNhdGVkIGluIHRoZSBjb2x1bW4gZGVzY3Jp -cHRvcg0KZm9yIHRoZSA8c3BhbiBjbGFzcz1TcGVsbEU+cG9zaXRpb25hbGx5PC9zcGFuPiBjb3Jy -ZXNwb25kaW5nIGNvbHVtbiBpbiB0aGUNCmV4cGxpY2l0IG9yIGltcGxpY2l0ICZsdDtpbnNlcnQg -Y29sdW1uIGxpc3QmZ3Q7LiBJZiA8aT5DVlMgPC9pPmlzIGFuICZsdDtlbXB0eQ0Kc3BlY2lmaWNh -dGlvbiZndDsgdGhhdCBzcGVjaWZpZXMgQVJSQVksIHRoZW4gPGk+RFQgPC9pPnNoYWxsIGJlIGFu -IGFycmF5IHR5cGUuDQpJZiA8aT5DVlMgPC9pPmlzIGFuICZsdDtlbXB0eSBzcGVjaWZpY2F0aW9u -Jmd0OyB0aGF0IHNwZWNpZmllcyBNVUxUSVNFVCwgdGhlbiA8aT5EVA0KPC9pPnNoYWxsIGJlIGEg -PHNwYW4gY2xhc3M9U3BlbGxFPm11bHRpc2V0PC9zcGFuPiB0eXBlLjxvOnA+PC9vOnA+PC9zcGFu -PjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246 -bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7 -Y29sb3I6YmxhY2snPjIwKSBFdmVyeSAmbHQ7bWVyZ2UgaW5zZXJ0IHZhbHVlDQplbGVtZW50Jmd0 -OyB3aG9zZSA8c3BhbiBjbGFzcz1TcGVsbEU+cG9zaXRpb25hbGx5PC9zcGFuPiBjb3JyZXNwb25k -aW5nDQombHQ7Y29sdW1uIG5hbWUmZ3Q7IGluICZsdDtpbnNlcnQgY29sdW1uIGxpc3QmZ3Q7IHJl -ZmVyZW5jZXMgYSBjb2x1bW4gb2Ygd2hpY2gNCnNvbWUgdW5kZXJseWluZyBjb2x1bW4gaXMgYSBn -ZW5lcmF0ZWQgY29sdW1uIHNoYWxsIGJlIGEgJmx0O2RlZmF1bHQgc3BlY2lmaWNhdGlvbiZndDsu -PG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1s -YXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0n -Zm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+MjEpIEZvciAxIChvbmUpIDwvc3Bhbj48c3Bh -bg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6U3ltYm9sO21zby1iaWRpLWZv -bnQtZmFtaWx5OlN5bWJvbDsNCmNvbG9yOmJsYWNrJz4mIzg4MDQ7IDwvc3Bhbj48c3BhbiBjbGFz -cz1TcGVsbEU+PGk+PHNwYW4gc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7DQpjb2xvcjpibGFjayc+ -aTwvc3Bhbj48L2k+PC9zcGFuPjxpPjxzcGFuIHN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9y -OmJsYWNrJz4NCk5JPC9zcGFuPjwvaT48c3BhbiBzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xv -cjpibGFjayc+LCB0aGUgU3ludGF4IFJ1bGVzIG9mIDwvc3Bhbj48c3Bhbg0KY2xhc3M9U3BlbGxF -PjxzcGFuIHN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOiMwMDAwNzAnPlN1YmNsYXVzZTwv -c3Bhbj48L3NwYW4+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOiMwMDAwNzAn -PiA5LjIsIJNTdG9yZSBhc3NpZ25tZW50lDwvc3Bhbj48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZTox -MS4wcHQ7Y29sb3I6YmxhY2snPiwgYXBwbHkgdG8gdGhlIGNvbHVtbiBvZiB0YWJsZSA8aT5UIDwv -aT5pZGVudGlmaWVkDQpieSB0aGUgPHNwYW4gY2xhc3M9U3BlbGxFPjxpPmk8L2k+LXRoPC9zcGFu -PiAmbHQ7Y29sdW1uIG5hbWUmZ3Q7IGluIHRoZQ0KJmx0O2luc2VydCBjb2x1bW4gbGlzdCZndDsg -YW5kIDxzcGFuIGNsYXNzPVNwZWxsRT48aT5FWFA8L2k+PGk+PHNwYW4NCnN0eWxlPSdmb250LXNp -emU6OS4wcHQnPmk8L3NwYW4+PC9pPjwvc3Bhbj48L3NwYW4+PGk+PHNwYW4gc3R5bGU9J2ZvbnQt -c2l6ZToNCjkuMHB0O2NvbG9yOmJsYWNrJz4gPC9zcGFuPjwvaT48c3BhbiBzdHlsZT0nZm9udC1z -aXplOjExLjBwdDtjb2xvcjpibGFjayc+YXMgPGk+VEFSR0VUDQo8L2k+YW5kIDxpPlZBTFVFPC9p -PiwgcmVzcGVjdGl2ZWx5LjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9y -bWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25l -Jz48Yj48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxNC4wcHQ7Y29sb3I6YmxhY2snPkFjY2VzcyBS -dWxlczxvOnA+PC9vOnA+PC9zcGFuPjwvYj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHls -ZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4N -CnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz4xKSBDYXNlOjxvOnA+PC9vOnA+ -PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQt -YWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZTox -MS4wcHQ7Y29sb3I6YmxhY2snPmEpIElmICZsdDttZXJnZSBzdGF0ZW1lbnQmZ3Q7IGlzDQpjb250 -YWluZWQsIHdpdGhvdXQgYW4gaW50ZXJ2ZW5pbmcgJmx0O1NRTCByb3V0aW5lIHNwZWMmZ3Q7IHRo -YXQgc3BlY2lmaWVzIFNRTCBTRUNVUklUWQ0KSU5WT0tFUiwgaW4gYW4gJmx0O1NRTCBzY2hlbWEg -c3RhdGVtZW50Jmd0OywgdGhlbiBsZXQgPGk+QSA8L2k+YmUgdGhlDQombHQ7YXV0aG9yaXphdGlv -biBpZGVudGlmaWVyJmd0OyB0aGF0IG93bnMgdGhhdCBzY2hlbWEuPG86cD48L286cD48L3NwYW4+ -PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpu -b25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpjbGFzcz1TcGVsbEU+PHNwYW4gc3R5bGU9 -J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPmk8L3NwYW4+PC9zcGFuPjxzcGFuDQpzdHls -ZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+KSBUaGUgYXBwbGljYWJsZSBwcml2aWxl -Z2VzIGZvciA8aT5BIDwvaT5zaGFsbA0KaW5jbHVkZSBVUERBVEUgZm9yIGVhY2ggdXBkYXRlIG9i -amVjdCBjb2x1bW4uPG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwg -c3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxz -cGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+aWkpIFRoZSBhcHBsaWNh -YmxlIHByaXZpbGVnZXMgZm9yIDxpPkEgPC9pPnNoYWxsDQppbmNsdWRlIElOU0VSVCBmb3IgZWFj -aCBpbnNlcnQgb2JqZWN0IGNvbHVtbi48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNz -PU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3Bh -Y2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz5paWkp -IElmICZsdDt0YXJnZXQgdGFibGUmZ3Q7IGltbWVkaWF0ZWx5DQpjb250YWlucyBPTkxZLCB0aGVu -IHRoZSBhcHBsaWNhYmxlIHByaXZpbGVnZXMgZm9yIDxpPkEgPC9pPnNoYWxsIGluY2x1ZGUgU0VM -RUNUDQpXSVRIIEhJRVJBUkNIWSBPUFRJT04gb24gYXQgbGVhc3Qgb25lIDxzcGFuIGNsYXNzPVNw -ZWxsRT5zdXBlcnRhYmxlPC9zcGFuPiBvZiA8aT5UPC9pPi48bzpwPjwvbzpwPjwvc3Bhbj48L3A+ -DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7 -dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9y -OmJsYWNrJz5iKSBPdGhlcndpc2UsPG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1N -c29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNl -Om5vbmUnPjxzcGFuDQpjbGFzcz1TcGVsbEU+PHNwYW4gc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7 -Y29sb3I6YmxhY2snPmk8L3NwYW4+PC9zcGFuPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBw -dDtjb2xvcjpibGFjayc+KSBUaGUgY3VycmVudCBwcml2aWxlZ2VzIHNoYWxsIGluY2x1ZGUNClVQ -REFURSBmb3IgZWFjaCB1cGRhdGUgb2JqZWN0IGNvbHVtbi48bzpwPjwvbzpwPjwvc3Bhbj48L3A+ -DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7 -dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9y -OmJsYWNrJz5paSkgVGhlIGN1cnJlbnQgcHJpdmlsZWdlcyBzaGFsbCBpbmNsdWRlDQpJTlNFUlQg -Zm9yIGVhY2ggaW5zZXJ0IG9iamVjdCBjb2x1bW4uPG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8 -cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQt -YXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFj -ayc+aWlpKSBJZiAmbHQ7dGFyZ2V0IHRhYmxlJmd0OyBpbW1lZGlhdGVseQ0KY29udGFpbnMgT05M -WSwgdGhlbiB0aGUgY3VycmVudCBwcml2aWxlZ2VzIHNoYWxsIGluY2x1ZGUgU0VMRUNUIFdJVEgg -SElFUkFSQ0hZDQpPUFRJT04gb24gYXQgbGVhc3Qgb25lIDxzcGFuIGNsYXNzPVNwZWxsRT5zdXBl -cnRhYmxlPC9zcGFuPiBvZiA8aT5UPC9pPi48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNs -YXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRv -c3BhY2U6bm9uZSc+PGI+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTQuMHB0O2NvbG9yOmJsYWNr -Jz5HZW5lcmFsIFJ1bGVzPG86cD48L286cD48L3NwYW4+PC9iPjwvcD4NCg0KPHAgY2xhc3M9TXNv -Tm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpu -b25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPjEpIElmIHRo -ZSBhY2Nlc3MgbW9kZSBvZiB0aGUgY3VycmVudA0KU1FMLXRyYW5zYWN0aW9uIG9yIHRoZSBhY2Nl -c3MgbW9kZSBvZiB0aGUgYnJhbmNoIG9mIHRoZSBjdXJyZW50IDxzcGFuDQpjbGFzcz1TcGVsbEU+ -U1FMdHJhbnNhY3Rpb248L3NwYW4+IGF0IHRoZSBjdXJyZW50IFNRTC1jb25uZWN0aW9uIGlzIHJl -YWQtb25seSwNCmFuZCA8aT5UIDwvaT5pcyBub3QgYSB0ZW1wb3JhcnkgdGFibGUsIHRoZW4gYW4g -ZXhjZXB0aW9uIGNvbmRpdGlvbiBpcyByYWlzZWQ6IDxpPmludmFsaWQNCnRyYW5zYWN0aW9uIHN0 -YXRlIJcgcmVhZC1vbmx5IFNRTC10cmFuc2FjdGlvbjwvaT4uPG86cD48L286cD48L3NwYW4+PC9w -Pg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25l -O3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xv -cjpibGFjayc+MikgSWYgdGhlcmUgaXMgYW55IHNlbnNpdGl2ZSBjdXJzb3IgPGk+Q1IgPC9pPnRo -YXQNCmlzIGN1cnJlbnRseSBvcGVuIGluIHRoZSBTUUwtdHJhbnNhY3Rpb24gaW4gd2hpY2ggdGhp -cyBTUUwtc3RhdGVtZW50IGlzIGJlaW5nDQpleGVjdXRlZCwgdGhlbjxvOnA+PC9vOnA+PC9zcGFu -PjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246 -bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7 -Y29sb3I6YmxhY2snPkNhc2U6PG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29O -b3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5v -bmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+YSkgSWYgPGk+ -Q1IgPC9pPmhhcyBub3QgYmVlbiBoZWxkIGludG8gYQ0Kc3Vic2VxdWVudCBTUUwtdHJhbnNhY3Rp -b24sIHRoZW4gZWl0aGVyIHRoZSBjaGFuZ2UgcmVzdWx0aW5nIGZyb20gdGhlIHN1Y2Nlc3NmdWwN -CmV4ZWN1dGlvbiBvZiB0aGlzIHN0YXRlbWVudCBzaGFsbCBiZSBtYWRlIHZpc2libGUgdG8gPGk+ -Q1IgPC9pPm9yIGFuIGV4Y2VwdGlvbg0KY29uZGl0aW9uIGlzIHJhaXNlZDo8bzpwPjwvbzpwPjwv -c3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFs -aWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCmNsYXNzPUdyYW1FPjxpPjxzcGFu -IHN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz5jdXJzb3I8L3NwYW4+PC9pPjwv -c3Bhbj48aT48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPiBzZW5z -aXRpdml0eSBleGNlcHRpb24glyByZXF1ZXN0IGZhaWxlZDwvc3Bhbj48L2k+PHNwYW4NCnN0eWxl -PSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz4uPG86cD48L286cD48L3NwYW4+PC9wPg0K -DQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3Rl -eHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpi -bGFjayc+YikgT3RoZXJ3aXNlLCB3aGV0aGVyIHRoZSBjaGFuZ2UgcmVzdWx0aW5nDQpmcm9tIHRo -ZSBzdWNjZXNzZnVsIGV4ZWN1dGlvbiBvZiB0aGlzIFNRTC1zdGF0ZW1lbnQgaXMgbWFkZSB2aXNp -YmxlIHRvIDxpPkNSIDwvaT5pcw0KaW1wbGVtZW50YXRpb24tZGVmaW5lZC48bzpwPjwvbzpwPjwv -c3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFs -aWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEu -MHB0O2NvbG9yOmJsYWNrJz4zKSBJZiB0aGVyZSBpcyBhbnkgY3Vyc29yIDxpPkNSIDwvaT50aGF0 -DQppcyBjdXJyZW50bHkgPHNwYW4gY2xhc3M9R3JhbUU+b3Blbjwvc3Bhbj4gYW5kIHdob3NlICZs -dDtkZWNsYXJlIGN1cnNvciZndDsNCmNvbnRhaW5lZCBJTlNFTlNJVElWRSwgdGhlbiBlaXRoZXIg -dGhlIGNoYW5nZSByZXN1bHRpbmcgZnJvbSB0aGUgc3VjY2Vzc2Z1bA0KZXhlY3V0aW9uIG9mIHRo -aXMgc3RhdGVtZW50IHNoYWxsIGJlIGludmlzaWJsZSB0byA8aT5DUjwvaT4sIG9yIGFuIGV4Y2Vw -dGlvbg0KY29uZGl0aW9uIGlzIHJhaXNlZDogPGk+Y3Vyc29yIHNlbnNpdGl2aXR5IGV4Y2VwdGlv -biCXIHJlcXVlc3QgZmFpbGVkPC9pPi48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNz -PU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3Bh -Y2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz40KSBU -aGUgZXh0ZW50IHRvIHdoaWNoIGFuDQpTUUwtaW1wbGVtZW50YXRpb24gbWF5IGRpc2FsbG93IGlu -ZGVwZW5kZW50IGNoYW5nZXMgdGhhdCBhcmUgbm90IHNpZ25pZmljYW50IGlzDQppbXBsZW1lbnRh -dGlvbi1kZWZpbmVkLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFs -IHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48 -c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPjUpIExldCA8aT5RVCA8 -L2k+YmUgdGhlIHRhYmxlIHNwZWNpZmllZCBieQ0KdGhlICZsdDt0YWJsZSByZWZlcmVuY2UmZ3Q7 -LiA8aT5RVCA8L2k+aXMgZWZmZWN0aXZlbHkgZXZhbHVhdGVkIGJlZm9yZSB1cGRhdGUNCm9yIGlu -c2VydGlvbiBvZiBhbnkgcm93cyBpbiA8aT5UPC9pPi4gTGV0IDxpPlEgPC9pPmJlIHRoZSByZXN1 -bHQgb2YgZXZhbHVhdGluZyA8aT5RVDwvaT4uPG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBj -bGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0 -b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+ -NikgRm9yIGVhY2ggJmx0O21lcmdlIHdoZW4gY2xhdXNlJmd0Oyw8bzpwPjwvbzpwPjwvc3Bhbj48 -L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5v -bmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2Nv -bG9yOmJsYWNrJz5DYXNlOjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9y -bWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25l -Jz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPmEpIElmICZsdDtt -ZXJnZSB3aGVuIG1hdGNoZWQgY2xhdXNlJmd0OyBpcw0Kc3BlY2lmaWVkLCB0aGVuOjxvOnA+PC9v -OnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdy -aWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0KY2xhc3M9U3BlbGxFPjxz -cGFuIHN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz5pPC9zcGFuPjwvc3Bhbj48 -c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPikgRm9yIGVhY2ggcm93 -IDxpPlIxIDwvaT5vZiA8aT5UPC9pPjo8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNz -PU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3Bh -Y2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz4xKSBU -aGUgJmx0O3NlYXJjaCBjb25kaXRpb24mZ3Q7IGlzIGFwcGxpZWQNCnRvIDxpPlIxIDwvaT53aXRo -IHRoZSBleHBvc2VkICZsdDt0YWJsZSBuYW1lJmd0OyBvZiB0aGUgJmx0O3RhcmdldCB0YWJsZSZn -dDsgYm91bmQNCnRvIDxpPlIxIDwvaT5hbmQgdG8gZWFjaCByb3cgb2YgPGk+USA8L2k+d2l0aCB0 -aGUgZXhwb3NlZCAmbHQ7Y29ycmVsYXRpb24NCm5hbWUmZ3Q7cyBvciAmbHQ7dGFibGUgb3IgcXVl -cnkgbmFtZSZndDtzIG9mIHRoZSAmbHQ7dGFibGUgcmVmZXJlbmNlJmd0OyBib3VuZA0KdG8gdGhh -dCByb3cuIFRoZSAmbHQ7c2VhcmNoIGNvbmRpdGlvbiZndDsgaXMgZWZmZWN0aXZlbHkgZXZhbHVh -dGVkIGZvciA8aT5SMSA8L2k+YmVmb3JlDQp1cGRhdGluZyBhbnkgcm93IG9mIDxpPlQgPC9pPmFu -ZCBwcmlvciB0byB0aGUgaW52b2NhdGlvbiBvZiBhbnkgJmx0O3RyaWdnZXJlZCBhY3Rpb24mZ3Q7 -DQpjYXVzZWQgYnkgdGhlIHVwZGF0ZSBvZiBhbnkgcm93IG9mIDxpPlQgPC9pPmFuZCBiZWZvcmUg -aW5zZXJ0aW5nIGFueSByb3dzIGludG8gPGk+VA0KPC9pPmFuZCBwcmlvciB0byB0aGUgaW52b2Nh -dGlvbiBvZiBhbnkgJmx0O3RyaWdnZXJlZCBhY3Rpb24mZ3Q7IGNhdXNlZCBieSB0aGUNCmluc2Vy -dCBvZiBhbnkgcm93IG9mIDxpPlQ8L2k+LiBFYWNoICZsdDs8c3BhbiBjbGFzcz1TcGVsbEU+c3Vi -cXVlcnk8L3NwYW4+Jmd0Ow0KaW4gdGhlICZsdDtzZWFyY2ggY29uZGl0aW9uJmd0OyBpcyBlZmZl -Y3RpdmVseSBleGVjdXRlZCBmb3IgPGk+UjEgPC9pPmFuZCBmb3INCmVhY2ggcm93IG9mIDxpPlEg -PC9pPmFuZCB0aGUgcmVzdWx0cyB1c2VkIGluIHRoZSBhcHBsaWNhdGlvbiBvZiB0aGUgJmx0O3Nl -YXJjaA0KY29uZGl0aW9uJmd0OyB0byA8aT5SMSA8L2k+YW5kIHRoZSBnaXZlbiByb3cgb2YgPGk+ -UTwvaT4uIElmIGFueSBleGVjdXRlZCAmbHQ7PHNwYW4NCmNsYXNzPVNwZWxsRT5zdWJxdWVyeTwv -c3Bhbj4mZ3Q7IGNvbnRhaW5zIGFuIG91dGVyIHJlZmVyZW5jZSB0byBhIGNvbHVtbiBvZiA8aT5U -PC9pPiwNCnRoZW4gdGhlIHJlZmVyZW5jZSBpcyB0byB0aGUgdmFsdWUgb2YgdGhhdCBjb2x1bW4g -aW4gdGhlIGdpdmVuIHJvdyBvZiA8aT5UPC9pPi48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxw -IGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1h -dXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNr -Jz5DYXNlOjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxl -PSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0K -c3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPkEpIElmICZsdDt0YXJnZXQgdGFi -bGUmZ3Q7IGNvbnRhaW5zIE9OTFksDQp0aGVuIDxpPlIxIDwvaT5pcyBhIHN1YmplY3Qgcm93IGlm -IDxpPlIxIDwvaT5oYXMgbm8gPHNwYW4gY2xhc3M9U3BlbGxFPnN1YnJvdzwvc3Bhbj4NCmluIGEg -cHJvcGVyIDxzcGFuIGNsYXNzPVNwZWxsRT5zdWJ0YWJsZTwvc3Bhbj4gb2YgPGk+VCA8L2k+YW5k -IHRoZSByZXN1bHQgb2YNCnRoZSAmbHQ7c2VhcmNoIGNvbmRpdGlvbiZndDsgaXMgPGk+VHJ1ZSA8 -L2k+Zm9yIHNvbWUgcm93IDxpPlIyIDwvaT5vZiA8aT5RPC9pPi4NCjxpPlIyIDwvaT5pcyB0aGUg -bWF0Y2hpbmcgcm93LjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFs -IHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48 -c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPkIpIE90aGVyd2lzZSwg -PGk+UjEgPC9pPmlzIGEgc3ViamVjdCByb3cNCmlmIHRoZSByZXN1bHQgb2YgdGhlICZsdDtzZWFy -Y2ggY29uZGl0aW9uJmd0OyBpcyA8aT5UcnVlIDwvaT5mb3Igc29tZSByb3cgPGk+UjINCjwvaT5v -ZiA8aT5RPC9pPi4gPGk+UjIgPC9pPmlzIHRoZSBtYXRjaGluZyByb3cuPG86cD48L286cD48L3Nw -YW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGln -bjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjkuMHB0 -O2NvbG9yOmJsYWNrJz5OT1RFIDM4MCCXIJNvdXRlciByZWZlcmVuY2WUIGlzIGRlZmluZWQgaW4g -PC9zcGFuPjxzcGFuDQpjbGFzcz1TcGVsbEU+PHNwYW4gc3R5bGU9J2ZvbnQtc2l6ZTo5LjBwdDtj -b2xvcjojMDAwMDcwJz5TdWJjbGF1c2U8L3NwYW4+PC9zcGFuPjxzcGFuDQpzdHlsZT0nZm9udC1z -aXplOjkuMHB0O2NvbG9yOiMwMDAwNzAnPiA2LjcsIJMmbHQ7Y29sdW1uIHJlZmVyZW5jZSZndDuU -PC9zcGFuPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjkuMHB0O2NvbG9yOmJsYWNrJz4uPG86cD48 -L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQt -Z3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1z -aXplOjExLjBwdDtjb2xvcjpibGFjayc+MikgSWYgPGk+UjEgPC9pPmlzIGEgc3ViamVjdCByb3cs -IHRoZW46PG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9 -J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpz -dHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+QSkgTGV0IDxpPk0gPC9pPmJlIHRo -ZSBudW1iZXIgb2YgbWF0Y2hpbmcNCnJvd3MgaW4gPGk+USA8L2k+Zm9yIDxpPlIxPC9pPi48bzpw -PjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91 -dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250 -LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz5CKSBJZiA8aT5NIDwvaT5pcyBncmVhdGVyIHRoYW4g -MSAob25lKSwNCnRoZW4gYW4gZXhjZXB0aW9uIGNvbmRpdGlvbiBpcyByYWlzZWQ6IDxpPmNhcmRp -bmFsaXR5IHZpb2xhdGlvbjwvaT4uPG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1N -c29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNl -Om5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+QykgVGhl -ICZsdDt1cGRhdGUgc291cmNlJmd0OyBvZiBlYWNoDQombHQ7c2V0IGNsYXVzZSZndDsgaXMgZWZm -ZWN0aXZlbHkgZXZhbHVhdGVkIGZvciA8aT5SMSA8L2k+YmVmb3JlIGFueSByb3cgb2YgPGk+VA0K -PC9pPmlzIHVwZGF0ZWQgYW5kIHByaW9yIHRvIHRoZSBpbnZvY2F0aW9uIG9mIGFueSAmbHQ7dHJp -Z2dlcmVkIGFjdGlvbiZndDsNCmNhdXNlZCBieSB0aGUgdXBkYXRlIG9mIGFueSByb3cgb2YgPGk+ -VDwvaT4uIFRoZSByZXN1bHRpbmcgdmFsdWUgaXMgdGhlIHVwZGF0ZQ0KdmFsdWUuPG86cD48L286 -cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3Jp -ZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXpl -OjExLjBwdDtjb2xvcjpibGFjayc+RCkgQSBjYW5kaWRhdGUgbmV3IHJvdyBpcyBjb25zdHJ1Y3Rl -ZCBieQ0KY29weWluZyB0aGUgc3ViamVjdCByb3cgYW5kIHVwZGF0aW5nIGl0IGFzIHNwZWNpZmll -ZCBieSBlYWNoICZsdDtzZXQgY2xhdXNlJmd0Ow0KYnkgYXBwbHlpbmcgdGhlIEdlbmVyYWwgUnVs -ZXMgb2YgPC9zcGFuPjxzcGFuIGNsYXNzPVNwZWxsRT48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZTox -MS4wcHQ7Y29sb3I6IzAwMDA3MCc+U3ViY2xhdXNlPC9zcGFuPjwvc3Bhbj48c3Bhbg0Kc3R5bGU9 -J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6IzAwMDA3MCc+IDE0LjEyLCCTJmx0O3NldCBjbGF1c2Ug -bGlzdCZndDuUPC9zcGFuPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFj -ayc+LjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdt -c28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5 -bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPmlpKSBJZiA8aT5UIDwvaT5pcyBhIGJh -c2UgdGFibGUsIHRoZW4gZWFjaA0Kc3ViamVjdCByb3cgaXMgYWxzbyBhbiBvYmplY3Qgcm93OyBv -dGhlcndpc2UsIGFuIG9iamVjdCByb3cgaXMgYW55IHJvdyBvZiBhDQpsZWFmIGdlbmVyYWxseSB1 -bmRlcmx5aW5nIHRhYmxlIG9mIDxpPlQgPC9pPmZyb20gd2hpY2ggYSBzdWJqZWN0IHJvdyBpcw0K -ZGVyaXZlZC48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHls -ZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4N -CnN0eWxlPSdmb250LXNpemU6OS4wcHQ7Y29sb3I6YmxhY2snPk5PVEUgMzgxIJcgPHNwYW4gY2xh -c3M9R3JhbUU+VGhlPC9zcGFuPg0KZGF0YSB2YWx1ZXMgYWxsb3dhYmxlIGluIHRoZSBvYmplY3Qg -cm93cyBtYXkgYmUgY29uc3RyYWluZWQgYnkgYSBXSVRIIENIRUNLDQpPUFRJT04gY29uc3RyYWlu -dC4gVGhlIGVmZmVjdCBvZiBhIFdJVEggQ0hFQ0sgT1BUSU9OIGNvbnN0cmFpbnQgaXMgZGVmaW5l -ZCBpbg0KdGhlIEdlbmVyYWwgUnVsZXMgb2YgPC9zcGFuPjxzcGFuIGNsYXNzPVNwZWxsRT48c3Bh -biBzdHlsZT0nZm9udC1zaXplOjkuMHB0Ow0KY29sb3I6IzAwMDA3MCc+U3ViY2xhdXNlPC9zcGFu -Pjwvc3Bhbj48c3BhbiBzdHlsZT0nZm9udC1zaXplOjkuMHB0O2NvbG9yOiMwMDAwNzAnPg0KMTQu -MjQsIJNFZmZlY3Qgb2YgcmVwbGFjaW5nIHNvbWUgcm93cyBpbiBhIHZpZXdlZCB0YWJsZZQ8L3Nw -YW4+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6OS4wcHQ7Y29sb3I6YmxhY2snPi48bzpwPjwvbzpw -Pjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlk -LWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6 -MTEuMHB0O2NvbG9yOmJsYWNrJz5paWkpIElmIGFueSByb3cgaW4gdGhlIHNldCBvZiBvYmplY3Qg -cm93cw0KaGFzIGJlZW4gbWFya2VkIGZvciBkZWxldGlvbiBieSBhbnkgJmx0O2RlbGV0ZSBzdGF0 -ZW1lbnQ6IHBvc2l0aW9uZWQmZ3Q7IHRoYXQNCmlkZW50aWZpZXMgc29tZSBjdXJzb3IgPGk+Q1Ig -PC9pPnRoYXQgaXMgc3RpbGwgb3BlbiBvciB1cGRhdGVkIGJ5IGFueQ0KJmx0O3VwZGF0ZSBzdGF0 -ZW1lbnQ6IHBvc2l0aW9uZWQmZ3Q7IHRoYXQgaWRlbnRpZmllcyBzb21lIGN1cnNvciA8aT5DUiA8 -L2k+dGhhdA0KaXMgc3RpbGwgb3BlbiwgdGhlbiBhIGNvbXBsZXRpb24gY29uZGl0aW9uIGlzIHJh -aXNlZDogPGk+d2FybmluZyCXIGN1cnNvcg0Kb3BlcmF0aW9uIGNvbmZsaWN0PC9pPi48bzpwPjwv -bzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1n -cmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCmNsYXNzPUdyYW1FPjxz -cGFuIHN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz5pdikgTGV0PC9zcGFuPjwv -c3Bhbj48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPiA8aT5DTCA8 -L2k+YmUgdGhlIGNvbHVtbnMgb2YgPGk+VCA8L2k+aWRlbnRpZmllZA0KYnkgdGhlICZsdDtvYmpl -Y3QgY29sdW1uJmd0O3MgY29udGFpbmVkIGluIHRoZSAmbHQ7c2V0IGNsYXVzZSBsaXN0Jmd0Oy48 -bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxh -eW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdm -b250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz52KSBFYWNoIHN1YmplY3Qgcm93IDxpPlNSIDwv -aT5pcw0KaWRlbnRpZmllZCBmb3IgcmVwbGFjZW1lbnQsIGJ5IGl0cyBjb3JyZXNwb25kaW5nIGNh -bmRpZGF0ZSBuZXcgcm93IDxpPkNOUjwvaT4sIGluDQo8aT5UPC9pPi4gVGhlIHNldCBvZiAoPGk+ -U1I8L2k+LCA8aT5DTlI8L2k+KSA8c3BhbiBjbGFzcz1HcmFtRT5wYWlycyBpczwvc3Bhbj4NCnRo -ZSByZXBsYWNlbWVudCBzZXQgZm9yIDxpPlQ8L2k+LjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0K -PHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0 -LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZTo5LjBwdDtjb2xvcjpibGFj -ayc+Tk9URSAzODIglyBJZGVudGlmeWluZyBhIHJvdyBmb3INCnJlcGxhY2VtZW50LCBhc3NvY2lh -dGluZyBhIHJlcGxhY2VtZW50IHJvdyB3aXRoIGFuIGlkZW50aWZpZWQgcm93LCBhbmQNCmFzc29j -aWF0aW5nIGEgcmVwbGFjZW1lbnQgc2V0IHdpdGggYSB0YWJsZSBhcmUgaW1wbGVtZW50YXRpb24t -ZGVwZW5kZW50DQpvcGVyYXRpb25zLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9 -TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFj -ZTpub25lJz48c3Bhbg0KY2xhc3M9R3JhbUU+PHNwYW4gc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7 -Y29sb3I6YmxhY2snPnZpKSBDYXNlPC9zcGFuPjwvc3Bhbj48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6 -ZToxMS4wcHQ7Y29sb3I6YmxhY2snPjo8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNz -PU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3Bh -Y2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz4xKSBJ -ZiA8aT5UIDwvaT5pcyBhIGJhc2UgdGFibGUsIHRoZW46PG86cD48L286cD48L3NwYW4+PC9wPg0K -DQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3Rl -eHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpi -bGFjayc+QSkgQ2FzZTo8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1h -bCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+ -PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz5JKSBJZiAmbHQ7dGFy -Z2V0IHRhYmxlJmd0OyBzcGVjaWZpZXMgT05MWSwNCnRoZW4gPGk+VCA8L2k+aXMgaWRlbnRpZmll -ZCBmb3IgcmVwbGFjZW1lbnQgcHJvY2Vzc2luZyB3aXRob3V0IDxzcGFuDQpjbGFzcz1TcGVsbEU+ -c3VidGFibGVzPC9zcGFuPiB3aXRoIHJlc3BlY3QgdG8gb2JqZWN0IGNvbHVtbnMgPGk+Q0w8L2k+ -LjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28t -bGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9 -J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPklJKSBPdGhlcndpc2UsIDxpPlQgPC9pPmlz -IGlkZW50aWZpZWQgZm9yDQpyZXBsYWNlbWVudCBwcm9jZXNzaW5nIHdpdGggPHNwYW4gY2xhc3M9 -U3BlbGxFPnN1YnRhYmxlczwvc3Bhbj4gd2l0aCByZXNwZWN0IHRvDQpvYmplY3QgY29sdW1ucyA8 -aT5DTDwvaT4uPG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5 -bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFu -DQpzdHlsZT0nZm9udC1zaXplOjkuMHB0O2NvbG9yOmJsYWNrJz5OT1RFIDM4MyCXIDxzcGFuIGNs -YXNzPUdyYW1FPklkZW50aWZ5aW5nPC9zcGFuPg0KYSBiYXNlIHRhYmxlIGZvciByZXBsYWNlbWVu -dCBwcm9jZXNzaW5nLCB3aXRoIG9yIHdpdGhvdXQgPHNwYW4gY2xhc3M9U3BlbGxFPnN1YnRhYmxl -czwvc3Bhbj4sDQppcyBhbiBpbXBsZW1lbnRhdGlvbi1kZXBlbmRlbnQgbWVjaGFuaXNtLiBJbiBn -ZW5lcmFsLCB0aG91Z2ggbm90IGhlcmUsIHRoZSBsaXN0DQpvZiBvYmplY3QgY29sdW1ucyBjYW4g -YmUgZW1wdHkuPG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5 -bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFu -DQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+QikgVGhlIEdlbmVyYWwgUnVs -ZXMgb2YgPC9zcGFuPjxzcGFuDQpjbGFzcz1TcGVsbEU+PHNwYW4gc3R5bGU9J2ZvbnQtc2l6ZTox -MS4wcHQ7Y29sb3I6IzAwMDA3MCc+U3ViY2xhdXNlPC9zcGFuPjwvc3Bhbj48c3Bhbg0Kc3R5bGU9 -J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6IzAwMDA3MCc+IDE0LjIyLCCTRWZmZWN0IG9mIHJlcGxh -Y2luZyByb3dzIGluDQpiYXNlIHRhYmxlc5Q8L3NwYW4+PHNwYW4gc3R5bGU9J2ZvbnQtc2l6ZTox -MS4wcHQ7Y29sb3I6YmxhY2snPiwgYXJlIGFwcGxpZWQuPG86cD48L286cD48L3NwYW4+PC9wPg0K -DQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3Rl -eHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpi -bGFjayc+MikgSWYgPGk+VCA8L2k+aXMgYSB2aWV3ZWQgdGFibGUsIHRoZW4gdGhlDQpHZW5lcmFs -IFJ1bGVzIG9mIDwvc3Bhbj48c3BhbiBjbGFzcz1TcGVsbEU+PHNwYW4gc3R5bGU9J2ZvbnQtc2l6 -ZToxMS4wcHQ7DQpjb2xvcjojMDAwMDcwJz5TdWJjbGF1c2U8L3NwYW4+PC9zcGFuPjxzcGFuIHN0 -eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOiMwMDAwNzAnPg0KMTQuMjQsIJNFZmZlY3Qgb2Yg -cmVwbGFjaW5nIHNvbWUgcm93cyBpbiBhIHZpZXdlZCB0YWJsZZQ8L3NwYW4+PHNwYW4NCnN0eWxl -PSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz4sIGFyZSBhcHBsaWVkIHdpdGggJmx0O3Rh -cmdldCB0YWJsZSZndDsgYXMNCjxpPlZJRVcgTkFNRTwvaT4uPG86cD48L286cD48L3NwYW4+PC9w -Pg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25l -O3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xv -cjpibGFjayc+YikgSWYgJmx0O21lcmdlIHdoZW4gbm90IG1hdGNoZWQNCmNsYXVzZSZndDsgaXMg -c3BlY2lmaWVkLCB0aGVuOjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9y -bWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25l -Jz48c3Bhbg0KY2xhc3M9U3BlbGxFPjxzcGFuIHN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9y -OmJsYWNrJz5pPC9zcGFuPjwvc3Bhbj48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29s -b3I6YmxhY2snPikgTGV0IDxpPlRSMSA8L2k+YmUgdGhlICZsdDt0YXJnZXQNCnRhYmxlJmd0OyBp -bW1lZGlhdGVseSBjb250YWluZWQgaW4gJmx0O21lcmdlIHN0YXRlbWVudCZndDssIGxldCA8aT5U -UjIgPC9pPmJlDQp0aGUgJmx0O3RhYmxlIHJlZmVyZW5jZSZndDsgaW1tZWRpYXRlbHkgY29udGFp -bmVkIGluICZsdDttZXJnZSBzdGF0ZW1lbnQmZ3Q7LA0KYW5kIGxldCA8aT5TQzEgPC9pPmJlIHRo -ZSAmbHQ7c2VhcmNoIGNvbmRpdGlvbiZndDsgaW1tZWRpYXRlbHkgY29udGFpbmVkIGluDQombHQ7 -bWVyZ2Ugc3RhdGVtZW50Jmd0Oy4gSWYgJmx0O21lcmdlIGNvcnJlbGF0aW9uIG5hbWUmZ3Q7IGlz -IHNwZWNpZmllZCwgbGV0IDxpPk1DTg0KPC9pPmJlIJNBUyAmbHQ7bWVyZ2UgY29ycmVsYXRpb24g -bmFtZSZndDuUOyBvdGhlcndpc2UsIGxldCA8aT5NQ04gPC9pPmJlIGENCnplcm8tbGVuZ3RoIHN0 -cmluZy4gTGV0IDxpPlMxIDwvaT5iZSB0aGUgcmVzdWx0IG9mPG86cD48L286cD48L3NwYW4+PC9w -Pg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25l -O3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjkuMHB0O2ZvbnQt -ZmFtaWx5OkNvdXJpZXI7bXNvLWJpZGktZm9udC1mYW1pbHk6Q291cmllcjsNCmNvbG9yOmJsYWNr -Jz5TRUxFQ1QgKjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0 -eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bh -bg0Kc3R5bGU9J2ZvbnQtc2l6ZTo5LjBwdDtmb250LWZhbWlseTpDb3VyaWVyO21zby1iaWRpLWZv -bnQtZmFtaWx5OkNvdXJpZXI7DQpjb2xvcjpibGFjayc+RlJPTSA8aT5UUjEgTUNOPC9pPiwgPGk+ -VFIyPG86cD48L286cD48L2k+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxl -PSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0K -c3R5bGU9J2ZvbnQtc2l6ZTo5LjBwdDtmb250LWZhbWlseTpDb3VyaWVyO21zby1iaWRpLWZvbnQt -ZmFtaWx5OkNvdXJpZXI7DQpjb2xvcjpibGFjayc+V0hFUkUgPGk+U0MxPG86cD48L286cD48L2k+ -PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQt -YWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZTox -MS4wcHQ7Y29sb3I6YmxhY2snPmlpKSBMZXQgPGk+UzIgPC9pPmJlIHRoZSBjb2xsZWN0aW9uIG9m -DQpyb3dzIG9mIDxpPlEgPC9pPmZvciB3aGljaCB0aGVyZSBleGlzdHMgaW4gPGk+UzEgPC9pPnNv -bWUgcm93IHRoYXQgaXMgdGhlDQpjb25jYXRlbmF0aW9uIG9mIHNvbWUgcm93IDxpPlIxIDwvaT5v -ZiA8aT5UIDwvaT5hbmQgc29tZSByb3cgPGk+UjIgPC9pPm9mIDxpPlE8L2k+LjxvOnA+PC9vOnA+ -PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQt -YWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZTox -MS4wcHQ7Y29sb3I6YmxhY2snPmlpaSkgTGV0IDxpPlMzIDwvaT5iZSB0aGUgY29sbGVjdGlvbiBv -Zg0Kcm93cyBvZiA8aT5RIDwvaT50aGF0IGFyZSBub3QgaW4gPGk+UzI8L2k+LiBMZXQgPGk+U04z -IDwvaT5iZSB0aGUgZWZmZWN0aXZlDQpkaXN0aW5jdCBuYW1lIGZvciA8aT5TMzwvaT4uIExldCA8 -aT5FTiA8L2k+YmUgdGhlIGV4cG9zZWQgJmx0O2NvcnJlbGF0aW9uDQpuYW1lJmd0OyBvciAmbHQ7 -dGFibGUgb3IgcXVlcnkgbmFtZSZndDsgb2YgPGk+VFIyPC9pPi48bzpwPjwvbzpwPjwvc3Bhbj48 -L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5v -bmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCmNsYXNzPUdyYW1FPjxzcGFuIHN0eWxlPSdm -b250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz5pdikgTGV0PC9zcGFuPjwvc3Bhbj48c3Bhbg0K -c3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPiA8aT5TNCA8L2k+YmUgdGhlIHJl -c3VsdCBvZjo8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHls -ZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4N -CnN0eWxlPSdmb250LXNpemU6OS4wcHQ7Zm9udC1mYW1pbHk6Q291cmllcjttc28tYmlkaS1mb250 -LWZhbWlseTpDb3VyaWVyOw0KY29sb3I6YmxhY2snPlNFTEVDVCA8aT5FWFA8L2k+PC9zcGFuPjxz -cGFuIHN0eWxlPSdmb250LXNpemU6Ny41cHQ7Zm9udC1mYW1pbHk6DQpDb3VyaWVyO21zby1iaWRp -LWZvbnQtZmFtaWx5OkNvdXJpZXI7Y29sb3I6YmxhY2snPjE8L3NwYW4+PHNwYW4NCnN0eWxlPSdm -b250LXNpemU6OS4wcHQ7Zm9udC1mYW1pbHk6Q291cmllcjttc28tYmlkaS1mb250LWZhbWlseTpD -b3VyaWVyOw0KY29sb3I6YmxhY2snPiwgPGk+RVhQPC9pPjwvc3Bhbj48c3BhbiBzdHlsZT0nZm9u -dC1zaXplOjcuNXB0O2ZvbnQtZmFtaWx5OkNvdXJpZXI7DQptc28tYmlkaS1mb250LWZhbWlseTpD -b3VyaWVyO2NvbG9yOmJsYWNrJz4yPC9zcGFuPjxzcGFuIHN0eWxlPSdmb250LXNpemU6OS4wcHQ7 -DQpmb250LWZhbWlseTpDb3VyaWVyO21zby1iaWRpLWZvbnQtZmFtaWx5OkNvdXJpZXI7Y29sb3I6 -YmxhY2snPiwgPHNwYW4NCmNsYXNzPUdyYW1FPi4uLiAsPC9zcGFuPiA8aT5FWFA8L2k+PC9zcGFu -PjxpPjxzcGFuIHN0eWxlPSdmb250LXNpemU6Ny41cHQ7DQpmb250LWZhbWlseTpDb3VyaWVyO21z -by1iaWRpLWZvbnQtZmFtaWx5OkNvdXJpZXI7Y29sb3I6YmxhY2snPk5JPG86cD48L286cD48L3Nw -YW4+PC9pPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQt -YWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZTo5 -LjBwdDtmb250LWZhbWlseTpDb3VyaWVyO21zby1iaWRpLWZvbnQtZmFtaWx5OkNvdXJpZXI7DQpj -b2xvcjpibGFjayc+RlJPTSA8aT5TTjMgPC9pPkFTIDxpPkVOPG86cD48L286cD48L2k+PC9zcGFu -PjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246 -bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7 -Y29sb3I6YmxhY2snPnYpIDxpPlM0IDwvaT5pcyBlZmZlY3RpdmVseSBldmFsdWF0ZWQNCmJlZm9y -ZSBpbnNlcnRpb24gb2YgYW55IHJvd3MgaW50byBvciB1cGRhdGUgb2YgYW55IHJvd3MgaW4gPGk+ -VDwvaT4uPG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9 -J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpj -bGFzcz1HcmFtRT48c3BhbiBzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+dmkp -IEZvcjwvc3Bhbj48L3NwYW4+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJs -YWNrJz4gZWFjaCByb3cgPGk+UiA8L2k+b2YgPGk+UzQ8L2k+OjxvOnA+PC9vOnA+PC9zcGFuPjwv -cD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9u -ZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29s -b3I6YmxhY2snPjEpIEEgY2FuZGlkYXRlIHJvdyBvZiA8aT5UIDwvaT5pcw0KZWZmZWN0aXZlbHkg -Y3JlYXRlZCBpbiB3aGljaCB0aGUgdmFsdWUgb2YgZWFjaCBjb2x1bW4gaXMgaXRzIGRlZmF1bHQg -dmFsdWUsIGFzDQpzcGVjaWZpZWQgaW4gdGhlIEdlbmVyYWwgUnVsZXMgb2YgPC9zcGFuPjxzcGFu -IGNsYXNzPVNwZWxsRT48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6IzAwMDA3 -MCc+U3ViY2xhdXNlPC9zcGFuPjwvc3Bhbj48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7 -Y29sb3I6IzAwMDA3MCc+IDExLjUsIJMmbHQ7ZGVmYXVsdCBjbGF1c2UmZ3Q7lDwvc3Bhbj48c3Bh -bg0Kc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPi4gVGhlIGNhbmRpZGF0ZSBy -b3cgY29uc2lzdHMgb2YgZXZlcnkNCmNvbHVtbiBvZiA8aT5UPC9pPi48bzpwPjwvbzpwPjwvc3Bh -bj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWdu -Om5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0 -O2NvbG9yOmJsYWNrJz4yKSBJZiA8aT5UIDwvaT5oYXMgYSBjb2x1bW4gPGk+UkMgPC9pPm9mDQp3 -aGljaCBzb21lIHVuZGVybHlpbmcgY29sdW1uIGlzIGEgc2VsZi1yZWZlcmVuY2luZyBjb2x1bW4s -IHRoZW4gPG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9 -J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpz -dHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+Q2FzZTo8bzpwPjwvbzpwPjwvc3Bh -bj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWdu -Om5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0 -O2NvbG9yOmJsYWNrJz5BKSBJZiA8aT5SQyA8L2k+aXMgYSBzeXN0ZW0tZ2VuZXJhdGVkDQpzZWxm -LXJlZmVyZW5jaW5nIGNvbHVtbiwgdGhlbiB0aGUgdmFsdWUgb2YgPGk+UkMgPC9pPmlzIGVmZmVj -dGl2ZWx5IHJlcGxhY2VkIGJ5DQp0aGUgUkVGIHZhbHVlIG9mIHRoZSBjYW5kaWRhdGUgcm93Ljxv -OnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5 -b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2Zv -bnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPkIpIElmIDxpPlJDIDwvaT5pcyBhIGRlcml2ZWQN -CnNlbGYtcmVmZXJlbmNpbmcgY29sdW1uLCB0aGVuIHRoZSB2YWx1ZSBvZiA8aT5SQyA8L2k+aXMg -ZWZmZWN0aXZlbHkgcmVwbGFjZWQgYnkNCmEgdmFsdWUgZGVyaXZlZCBmcm9tIHRoZSBjb2x1bW5z -IGluIHRoZSBjYW5kaWRhdGUgcm93IHRoYXQgY29ycmVzcG9uZCB0byB0aGUNCmxpc3Qgb2YgYXR0 -cmlidXRlcyBvZiB0aGUgZGVyaXZlZCByZXByZXNlbnRhdGlvbiBvZiB0aGUgcmVmZXJlbmNlIHR5 -cGUgb2YgPGk+UkMNCjwvaT5pbiBhbiBpbXBsZW1lbnRhdGlvbiBkZXBlbmRlbnQgbWFubmVyLjxv -OnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5 -b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9J2Zv -bnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPjMpIEZvciBlYWNoIG9iamVjdCBjb2x1bW4gaW4g -dGhlIGNhbmRpZGF0ZQ0Kcm93LCBsZXQgPHNwYW4gY2xhc3M9U3BlbGxFPjxpPkM8L2k+PGk+PHNw -YW4gc3R5bGU9J2ZvbnQtc2l6ZTo5LjBwdCc+aTwvc3Bhbj48L2k+PC9zcGFuPjwvc3Bhbj48aT48 -c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZTo5LjBwdDtjb2xvcjpibGFjayc+IDwvc3Bhbj48L2k+PHNw -YW4gc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7DQpjb2xvcjpibGFjayc+YmUgdGhlIG9iamVjdCBj -b2x1bW4gaWRlbnRpZmllZCBieSA8c3BhbiBjbGFzcz1HcmFtRT50aGUgPHNwYW4NCmNsYXNzPVNw -ZWxsRT48aT5pPC9pPjwvc3Bhbj48L3NwYW4+PHNwYW4gY2xhc3M9U3BlbGxFPi10aDwvc3Bhbj4g -Jmx0O2NvbHVtbg0KbmFtZSZndDsgaW4gdGhlICZsdDtpbnNlcnQgY29sdW1uIGxpc3QmZ3Q7IGFu -ZCBsZXQgPHNwYW4gY2xhc3M9U3BlbGxFPjxpPlNWPC9pPjxpPjxzcGFuDQpzdHlsZT0nZm9udC1z -aXplOjkuMHB0Jz5pPC9zcGFuPjwvaT48L3NwYW4+PC9zcGFuPjxpPjxzcGFuIHN0eWxlPSdmb250 -LXNpemU6DQo5LjBwdDtjb2xvcjpibGFjayc+IDwvc3Bhbj48L2k+PHNwYW4gc3R5bGU9J2ZvbnQt -c2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPmJlDQp0aGUgPHNwYW4gY2xhc3M9U3BlbGxFPjxpPmk8 -L2k+LXRoPC9zcGFuPiB2YWx1ZSBvZiA8aT5SPC9pPi48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoN -CjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4 -dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJs -YWNrJz40KSBGb3IgZXZlcnkgPHNwYW4gY2xhc3M9U3BlbGxFPjxpPkM8L2k+PGk+PHNwYW4NCnN0 -eWxlPSdmb250LXNpemU6OS4wcHQnPmk8L3NwYW4+PC9pPjwvc3Bhbj48L3NwYW4+PGk+PHNwYW4g -c3R5bGU9J2ZvbnQtc2l6ZToNCjkuMHB0O2NvbG9yOmJsYWNrJz4gPC9zcGFuPjwvaT48c3BhbiBz -dHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+Zm9yDQp3aGljaCBvbmUgb2YgdGhl -IGZvbGxvd2luZyBjb25kaXRpb25zIGlzIHRydWU6PG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8 -cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQt -YXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFj -ayc+QSkgPHNwYW4gY2xhc3M9U3BlbGxFPjxpPkM8L2k+PGk+PHNwYW4NCnN0eWxlPSdmb250LXNp -emU6OS4wcHQnPmk8L3NwYW4+PC9pPjwvc3Bhbj48L3NwYW4+PGk+PHNwYW4gc3R5bGU9J2ZvbnQt -c2l6ZToNCjkuMHB0O2NvbG9yOmJsYWNrJz4gPC9zcGFuPjwvaT48c3BhbiBzdHlsZT0nZm9udC1z -aXplOjExLjBwdDtjb2xvcjpibGFjayc+aXMNCm5vdCBtYXJrZWQgYXMgdW5hc3NpZ25lZCBhbmQg -bm8gdW5kZXJseWluZyBjb2x1bW4gb2YgPHNwYW4gY2xhc3M9U3BlbGxFPjxpPkM8L2k+PGk+PHNw -YW4NCnN0eWxlPSdmb250LXNpemU6OS4wcHQnPmk8L3NwYW4+PC9pPjwvc3Bhbj48L3NwYW4+PGk+ -PHNwYW4gc3R5bGU9J2ZvbnQtc2l6ZToNCjkuMHB0O2NvbG9yOmJsYWNrJz4gPC9zcGFuPjwvaT48 -c3BhbiBzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+aXMgYQ0Kc2VsZi1yZWZl -cmVuY2luZyBjb2x1bW4uPG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3Jt -YWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUn -PjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+QikgU29tZSB1bmRl -cmx5aW5nIGNvbHVtbiBvZiA8c3Bhbg0KY2xhc3M9U3BlbGxFPjxpPkM8L2k+PGk+PHNwYW4gc3R5 -bGU9J2ZvbnQtc2l6ZTo5LjBwdCc+aTwvc3Bhbj48L2k+PC9zcGFuPjwvc3Bhbj48aT48c3Bhbg0K -c3R5bGU9J2ZvbnQtc2l6ZTo5LjBwdDtjb2xvcjpibGFjayc+IDwvc3Bhbj48L2k+PHNwYW4gc3R5 -bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7DQpjb2xvcjpibGFjayc+aXMgYSB1c2VyLWdlbmVyYXRlZCBz -ZWxmLXJlZmVyZW5jaW5nIGNvbHVtbi48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNz -PU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3Bh -Y2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz5DKSBT -b21lIHVuZGVybHlpbmcgY29sdW1uIG9mIDxzcGFuDQpjbGFzcz1TcGVsbEU+PGk+QzwvaT48aT48 -c3BhbiBzdHlsZT0nZm9udC1zaXplOjkuMHB0Jz5pPC9zcGFuPjwvaT48L3NwYW4+PC9zcGFuPjxp -PjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjkuMHB0O2NvbG9yOmJsYWNrJz4gPC9zcGFuPjwvaT48 -c3BhbiBzdHlsZT0nZm9udC1zaXplOjExLjBwdDsNCmNvbG9yOmJsYWNrJz5pcyBhIHNlbGYtcmVm -ZXJlbmNpbmcgY29sdW1uIGFuZCBPVkVSUklESU5HIFNZU1RFTSBWQUxVRSBpcw0Kc3BlY2lmaWVk -LjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28t -bGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0Kc3R5bGU9 -J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPkQpIFNvbWUgdW5kZXJseWluZyBjb2x1bW4g -b2YgPHNwYW4NCmNsYXNzPVNwZWxsRT48aT5DPC9pPjxpPjxzcGFuIHN0eWxlPSdmb250LXNpemU6 -OS4wcHQnPmk8L3NwYW4+PC9pPjwvc3Bhbj48L3NwYW4+PGk+PHNwYW4NCnN0eWxlPSdmb250LXNp -emU6OS4wcHQ7Y29sb3I6YmxhY2snPiA8L3NwYW4+PC9pPjxzcGFuIHN0eWxlPSdmb250LXNpemU6 -MTEuMHB0Ow0KY29sb3I6YmxhY2snPmlzIGFuIGlkZW50aXR5IGNvbHVtbiBhbmQgT1ZFUlJJRElO -RyBTWVNURU0gVkFMVUUgaXMgc3BlY2lmaWVkLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAg -Y2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1 -dG9zcGFjZTpub25lJz48c3Bhbg0KY2xhc3M9R3JhbUU+PHNwYW4gc3R5bGU9J2ZvbnQtc2l6ZTox -MS4wcHQ7Y29sb3I6YmxhY2snPnRoZTwvc3Bhbj48L3NwYW4+PHNwYW4NCnN0eWxlPSdmb250LXNp -emU6MTEuMHB0O2NvbG9yOmJsYWNrJz4gR2VuZXJhbCBSdWxlcyBvZiA8L3NwYW4+PHNwYW4NCmNs -YXNzPVNwZWxsRT48c3BhbiBzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjojMDAwMDcwJz5T -dWJjbGF1c2U8L3NwYW4+PC9zcGFuPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xv -cjojMDAwMDcwJz4gOS4yLCCTU3RvcmUgYXNzaWdubWVudJQ8L3NwYW4+PHNwYW4NCnN0eWxlPSdm -b250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz4sIGFyZSBhcHBsaWVkIHRvIDxzcGFuIGNsYXNz -PVNwZWxsRT48aT5DPC9pPjxpPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjkuMHB0Jz5pPC9zcGFu -PjwvaT48L3NwYW4+PC9zcGFuPjxpPjxzcGFuIHN0eWxlPSdmb250LXNpemU6DQo5LjBwdDtjb2xv -cjpibGFjayc+IDwvc3Bhbj48L2k+PHNwYW4gc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6 -YmxhY2snPmFuZCA8c3Bhbg0KY2xhc3M9U3BlbGxFPjxpPlNWPC9pPjxpPjxzcGFuIHN0eWxlPSdm -b250LXNpemU6OS4wcHQnPmk8L3NwYW4+PC9pPjwvc3Bhbj48L3NwYW4+PGk+PHNwYW4NCnN0eWxl -PSdmb250LXNpemU6OS4wcHQ7Y29sb3I6YmxhY2snPiA8L3NwYW4+PC9pPjxzcGFuIHN0eWxlPSdm -b250LXNpemU6MTEuMHB0Ow0KY29sb3I6YmxhY2snPmFzIDxpPlRBUkdFVCA8L2k+YW5kIDxpPlNP -VVJDRTwvaT4sIHJlc3BlY3RpdmVseS48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNz -PU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3Bh -Y2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6OS4wcHQ7Y29sb3I6YmxhY2snPk5PVEUg -Mzg0IJcgPHNwYW4gY2xhc3M9R3JhbUU+VGhlPC9zcGFuPg0KZGF0YSB2YWx1ZXMgYWxsb3dhYmxl -IGluIHRoZSBjYW5kaWRhdGUgcm93IG1heSBiZSBjb25zdHJhaW5lZCBieSBhIFdJVEggQ0hFQ0sN -Ck9QVElPTiBjb25zdHJhaW50LiBUaGUgZWZmZWN0IG9mIGEgV0lUSCBDSEVDSyBPUFRJT04gY29u -c3RyYWludCBpcyBkZWZpbmVkIGluDQp0aGUgR2VuZXJhbCBSdWxlcyBvZiA8L3NwYW4+PHNwYW4g -Y2xhc3M9U3BlbGxFPjxzcGFuIHN0eWxlPSdmb250LXNpemU6OS4wcHQ7DQpjb2xvcjojMDAwMDcw -Jz5TdWJjbGF1c2U8L3NwYW4+PC9zcGFuPjxzcGFuIHN0eWxlPSdmb250LXNpemU6OS4wcHQ7Y29s -b3I6IzAwMDA3MCc+DQoxNC4yMSwgk0VmZmVjdCBvZiBpbnNlcnRpbmcgYSB0YWJsZSBpbnRvIGEg -dmlld2VkIHRhYmxllDwvc3Bhbj48c3Bhbg0Kc3R5bGU9J2ZvbnQtc2l6ZTo5LjBwdDtjb2xvcjpi -bGFjayc+LjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxl -PSdtc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTt0ZXh0LWF1dG9zcGFjZTpub25lJz48c3Bhbg0K -c3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6YmxhY2snPnZpaSkgTGV0IDxpPlMgPC9pPmJl -IHRoZSB0YWJsZSBjb25zaXN0aW5nDQpvZiB0aGUgY2FuZGlkYXRlIHJvd3MuPG86cD48L286cD48 -L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1h -bGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjEx -LjBwdDtjb2xvcjpibGFjayc+Q2FzZTo8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNz -PU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3Bh -Y2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz4xKSBJ -ZiA8aT5UIDwvaT5pcyBhIGJhc2UgdGFibGUsIHRoZW46PG86cD48L286cD48L3NwYW4+PC9wPg0K -DQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3Rl -eHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpi -bGFjayc+QSkgPGk+VCA8L2k+aXMgPGk+aWRlbnRpZmllZCBmb3IgaW5zZXJ0aW9uDQpvZiBzb3Vy -Y2UgdGFibGUgUzwvaT4uPG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3Jt -YWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUn -PjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjkuMHB0O2NvbG9yOmJsYWNrJz5OT1RFIDM4NSCXIDxz -cGFuIGNsYXNzPUdyYW1FPklkZW50aWZ5aW5nPC9zcGFuPg0KYSBiYXNlIHRhYmxlIGZvciBpbnNl -cnRpb24gb2YgYSBzb3VyY2UgdGFibGUgaXMgYW4gaW1wbGVtZW50YXRpb24tZGVwZW5kZW50IG9w -ZXJhdGlvbi48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHls -ZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4N -CnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz5CKSBUaGUgR2VuZXJhbCBSdWxl -cyBvZiA8L3NwYW4+PHNwYW4NCmNsYXNzPVNwZWxsRT48c3BhbiBzdHlsZT0nZm9udC1zaXplOjEx -LjBwdDtjb2xvcjojMDAwMDcwJz5TdWJjbGF1c2U8L3NwYW4+PC9zcGFuPjxzcGFuDQpzdHlsZT0n -Zm9udC1zaXplOjExLjBwdDtjb2xvcjojMDAwMDcwJz4gMTQuMTksIJNFZmZlY3Qgb2YgaW5zZXJ0 -aW5nIHRhYmxlcyBpbnRvDQpiYXNlIHRhYmxlc5Q8L3NwYW4+PHNwYW4gc3R5bGU9J2ZvbnQtc2l6 -ZToxMS4wcHQ7Y29sb3I6YmxhY2snPiwgYXJlIGFwcGxpZWQuPG86cD48L286cD48L3NwYW4+PC9w -Pg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25l -O3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xv -cjpibGFjayc+MikgSWYgPGk+VCA8L2k+aXMgYSB2aWV3ZWQgdGFibGUsIHRoZW4gdGhlDQpHZW5l -cmFsIFJ1bGVzIG9mIDwvc3Bhbj48c3BhbiBjbGFzcz1TcGVsbEU+PHNwYW4gc3R5bGU9J2ZvbnQt -c2l6ZToxMS4wcHQ7DQpjb2xvcjojMDAwMDcwJz5TdWJjbGF1c2U8L3NwYW4+PC9zcGFuPjxzcGFu -IHN0eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOiMwMDAwNzAnPg0KMTQuMjEsIJNFZmZlY3Qg -b2YgaW5zZXJ0aW5nIGEgdGFibGUgaW50byBhIHZpZXdlZCB0YWJsZZQ8L3NwYW4+PHNwYW4NCnN0 -eWxlPSdmb250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz4sIGFyZSBhcHBsaWVkIHdpdGggPGk+ -UyA8L2k+YXMgPGk+U09VUkNFIDwvaT5hbmQNCjxpPlQgPC9pPmFzIDxpPlRBUkdFVDwvaT4uPG86 -cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlv -dXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9u -dC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+NykgSWYgPGk+USA8L2k+aXMgZW1wdHksIHRoZW4g -YSBjb21wbGV0aW9uDQpjb25kaXRpb24gaXMgcmFpc2VkOiA8aT5ubyBkYXRhPC9pPi48bzpwPjwv -bzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1n -cmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PGI+PHNwYW4NCnN0eWxlPSdmb250 -LXNpemU6MTQuMHB0O2NvbG9yOmJsYWNrJz5Db25mb3JtYW5jZSBSdWxlczxvOnA+PC9vOnA+PC9z -cGFuPjwvYj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlk -LWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6 -MTEuMHB0O2NvbG9yOmJsYWNrJz4xKSBXaXRob3V0IEZlYXR1cmUgRjc4MSwgk1NlbGYtcmVmZXJl -bmNpbmcNCm9wZXJhdGlvbnOULCBjb25mb3JtaW5nIFNRTCBsYW5ndWFnZSBzaGFsbCBub3QgY29u -dGFpbiBhICZsdDttZXJnZSBzdGF0ZW1lbnQmZ3Q7DQppbiB3aGljaCBhIGxlYWYgZ2VuZXJhbGx5 -IHVuZGVybHlpbmcgdGFibGUgb2YgPGk+VCA8L2k+aXMgZ2VuZXJhbGx5IGNvbnRhaW5lZA0KaW4g -YSAmbHQ7cXVlcnkgZXhwcmVzc2lvbiZndDsgaW1tZWRpYXRlbHkgY29udGFpbmVkIGluIHRoZSAm -bHQ7dGFibGUgcmVmZXJlbmNlJmd0Ow0KZXhjZXB0IGFzIHRoZSAmbHQ7dGFibGUgb3IgcXVlcnkg -bmFtZSZndDsgb3IgJmx0O2NvcnJlbGF0aW9uIG5hbWUmZ3Q7IG9mIGENCmNvbHVtbiByZWZlcmVu -Y2UuPG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21z -by1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHls -ZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+MikgV2l0aG91dCBGZWF0dXJlIEY3ODEs -IJNTZWxmLXJlZmVyZW5jaW5nDQpvcGVyYXRpb25zlCwgY29uZm9ybWluZyBTUUwgbGFuZ3VhZ2Ug -c2hhbGwgbm90IGNvbnRhaW4gYSAmbHQ7bWVyZ2Ugc3RhdGVtZW50Jmd0Ow0KaW4gd2hpY2ggYSBs -ZWFmIGdlbmVyYWxseSB1bmRlcmx5aW5nIHRhYmxlIG9mIDxpPlQgPC9pPmlzIGFuIHVuZGVybHlp -bmcgdGFibGUNCm9mIGFueSAmbHQ7cXVlcnkgZXhwcmVzc2lvbiZndDsgZ2VuZXJhbGx5IGNvbnRh -aW5lZCBpbiB0aGUgJmx0O3NlYXJjaA0KY29uZGl0aW9uJmd0Oy48bzpwPjwvbzpwPjwvc3Bhbj48 -L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxheW91dC1ncmlkLWFsaWduOm5v -bmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdmb250LXNpemU6MTEuMHB0O2Nv -bG9yOmJsYWNrJz4zKSBXaXRob3V0IEZlYXR1cmUgUzAyNCwgk0VuaGFuY2VkDQpzdHJ1Y3R1cmVk -IHR5cGVzlCwgY29uZm9ybWluZyBTUUwgbGFuZ3VhZ2Ugc2hhbGwgbm90IGNvbnRhaW4gYSAmbHQ7 -bWVyZ2Ugc3RhdGVtZW50Jmd0Ow0KdGhhdCBkb2VzIG5vdCBzYXRpc2Z5IHRoZSBjb25kaXRpb246 -IGZvciBlYWNoIGNvbHVtbiA8aT5DIDwvaT5pZGVudGlmaWVkIGluIHRoZQ0KZXhwbGljaXQgb3Ig -aW1wbGljaXQgJmx0O2luc2VydCBjb2x1bW4gbGlzdCZndDssIGlmIHRoZSBkZWNsYXJlZCB0eXBl -IG9mIDxpPkMgPC9pPmlzDQphIHN0cnVjdHVyZWQgdHlwZSA8aT5UWTwvaT4sIHRoZW4gdGhlIGRl -Y2xhcmVkIHR5cGUgb2YgdGhlIGNvcnJlc3BvbmRpbmcgY29sdW1uDQpvZiB0aGUgJmx0O3F1ZXJ5 -IGV4cHJlc3Npb24mZ3Q7IG9yICZsdDtjb250ZXh0dWFsbHkgdHlwZWQgdGFibGUgdmFsdWUNCmNv -bnN0cnVjdG9yJmd0OyBpcyA8aT5UWTwvaT4uPG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBj -bGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lO3RleHQtYXV0 -b3NwYWNlOm5vbmUnPjxzcGFuDQpzdHlsZT0nZm9udC1zaXplOjExLjBwdDtjb2xvcjpibGFjayc+ -NCkgV2l0aG91dCBGZWF0dXJlIEYzMTIsIJNNRVJHRSBzdGF0ZW1lbnSULA0KY29uZm9ybWluZyBT -UUwgbGFuZ3VhZ2Ugc2hhbGwgbm90IGNvbnRhaW4gYSAmbHQ7bWVyZ2Ugc3RhdGVtZW50Jmd0Oy48 -bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxh -eW91dC1ncmlkLWFsaWduOm5vbmU7dGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4NCnN0eWxlPSdm -b250LXNpemU6MTEuMHB0O2NvbG9yOmJsYWNrJz41KSBXaXRob3V0IEZlYXR1cmUgVDExMSwgk1Vw -ZGF0YWJsZSBqb2lucywNCnVuaW9ucywgYW5kIGNvbHVtbnOULCBjb25mb3JtaW5nIFNRTCBsYW5n -dWFnZSBzaGFsbCBub3QgY29udGFpbiBhICZsdDttZXJnZQ0Kc3RhdGVtZW50Jmd0OyB0aGF0IGNv -bnRhaW5zIGFuICZsdDt0YXJnZXQgdGFibGUmZ3Q7IHRoYXQgaWRlbnRpZmllcyBhIHRhYmxlDQp0 -aGF0IGlzIG5vdCBzaW1wbHkgdXBkYXRhYmxlLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPC9k -aXY+DQoNCjwvYm9keT4NCg0KPC9odG1sPg0K - -------_=_NextPart_001_01C5EBF2.45ACB7DC-- - -From pgsql-hackers-owner+M76240@postgresql.org Fri Nov 18 00:37:18 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jAI5bHO24992 - for ; Fri, 18 Nov 2005 00:37:17 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 568F2C4BB5D; - Fri, 18 Nov 2005 05:37:13 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id B2ABDDB7D0 - for ; Fri, 18 Nov 2005 00:57:11 -0400 (AST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 41664-06 - for ; - Fri, 18 Nov 2005 04:57:10 +0000 (GMT) -X-Greylist: from auto-whitelisted by SQLgrey- -Received: from zigo.dhs.org (ua-83-227-204-174.cust.bredbandsbolaget.se [83.227.204.174]) - by svr1.postgresql.org (Postfix) with ESMTP id E7930DB745 - for ; Fri, 18 Nov 2005 00:57:08 -0400 (AST) -Received: from zigo.zigo.dhs.org (zigo.zigo.dhs.org [192.168.0.1]) - by zigo.dhs.org (Postfix) with ESMTP - id EC5488467; Fri, 18 Nov 2005 05:57:05 +0100 (CET) -Date: Fri, 18 Nov 2005 05:57:05 +0100 (CET) -From: Dennis Bjorklund -To: Bruce Momjian -cc: Zeugswetter Andreas DCP SD , - Csaba Nagy , - Martijn van Oosterhout , Dann Corbit , - Simon Riggs , - Rick Gigger , Tom Lane , - Christopher Kings-Lynne , - "Jim C. Nasby" , , - , Jaime Casanova , - Peter Eisentraut -Subject: Re: [HACKERS] MERGE vs REPLACE -In-Reply-To: <200511180218.jAI2IPF04976@candle.pha.pa.us> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=ISO-8859-1 -Content-Transfer-Encoding: 8BIT -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.479 required=5 tests=[DNS_FROM_RFC_ABUSE=0.479] -X-Spam-Score: 0.479 -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: ORr - -On Thu, 17 Nov 2005, Bruce Momjian wrote: - -> Unless you have a table lock, INSERT has to be before UPDATE, think -> UPDATE, UPDATE (both fail), INSERT, INSERT. - -No matter what operation you start with you need a loop that try -insert/update until one of them succeed like in this example: - -http://www.postgresql.org/docs/8.1/static/plpgsql-control-structures.html#PLPGSQL-UPSERT-EXAMPLE - -Without a loop you might not get to execute neither the insert nor the -update. Why? Think about this example: - -BEGIN - -INSERT <- fail because there is a row already - - <- before we manage to do the update someone - delete the row (which we can see in the - default transaction isolation level) - -UPDATE <- fail because there is no row so we will loop - and try the insert again - - <- before we manage to do the insert someone else does - an insert - -INSERT <- fail because there is a row already - - <- before we manage to do the update someone - delete the row -.... - - -You might need to loop any number of times before you manage to perform -one of the two operations. Which operation you should start with depends -on which of the two cases is the common one. - --- -/Dennis Björklund - - ----------------------------(end of broadcast)--------------------------- -TIP 4: Have you searched our list archives? - - http://archives.postgresql.org - -From ZeugswetterA@spardat.at Fri Nov 18 05:32:16 2005 -Return-path: -Received: from smxsat1.smxs.net (smxsat1.smxs.net [213.150.10.1]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jAIAWFO17231 - for ; Fri, 18 Nov 2005 05:32:15 -0500 (EST) -Received: from m01x1.s-mxs.net [10.3.55.201] - by smxsat1.smxs.net - over TLS secured channel - with XWall v3.35d. ; - Fri, 18 Nov 2005 11:32:04 +0100 -Received: from m0102.s-mxs.net [10.3.55.2] - by m01x1.s-mxs.net - with XWall v3.35d. ; - Fri, 18 Nov 2005 11:33:38 +0100 -Received: from m0143.s-mxs.net ([10.252.53.143]) by m0102.s-mxs.net with Microsoft SMTPSVC(6.0.3790.1830); - Fri, 18 Nov 2005 11:33:38 +0100 -X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 -content-class: urn:content-classes:message -Subject: RE: [HACKERS] MERGE vs REPLACE -MIME-Version: 1.0 -Content-Type: text/plain; - charset="us-ascii" -Date: Fri, 18 Nov 2005 11:32:01 +0100 -Message-ID: -Thread-Topic: [HACKERS] MERGE vs REPLACE -Thread-Index: AcXr5pYcJO3ZR4d/RwebJDquaJx6SQARC+uA -From: "Zeugswetter Andreas DCP SD" -To: "Bruce Momjian" -cc: "Csaba Nagy" , - "Martijn van Oosterhout" , - "Dann Corbit" , "Simon Riggs" , - "Rick Gigger" , "Tom Lane" , - "Christopher Kings-Lynne" , - "Jim C. Nasby" , , - , "Jaime Casanova" , - "Peter Eisentraut" -X-OriginalArrivalTime: 18 Nov 2005 10:33:38.0046 (UTC) FILETIME=[8941B1E0:01C5EC2B] -Content-Transfer-Encoding: 8bit -X-MIME-Autoconverted: from quoted-printable to 8bit by candle.pha.pa.us id jAIAWFO17231 -Status: OR - - -> Unless you have a table lock, INSERT has to be before UPDATE, think -UPDATE, UPDATE (both fail), INSERT, INSERT. - -> > update -> > if no rows updated -> > insert -> > if duplicate key -> > update -> > if no rows updated goto insert - -That is why you have the loop. This is not a problem with above code, -because only one insert succeeds -while the others then do the update. - -Andreas - -From pgsql-hackers-owner+M76266@postgresql.org Fri Nov 18 13:23:17 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jAIINGO19600 - for ; Fri, 18 Nov 2005 13:23:17 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 33CABC4BB56; - Fri, 18 Nov 2005 18:23:14 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 54076DBA92 - for ; Fri, 18 Nov 2005 11:35:26 -0400 (AST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 41216-03 - for ; - Fri, 18 Nov 2005 15:35:29 +0000 (GMT) -X-Greylist: from auto-whitelisted by SQLgrey- -Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) - by svr1.postgresql.org (Postfix) with ESMTP id 84C42DBAAE - for ; Fri, 18 Nov 2005 11:35:23 -0400 (AST) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id jAIFYZcu006411; - Fri, 18 Nov 2005 10:34:35 -0500 (EST) -To: Bruce Momjian -cc: Dennis Bjorklund , - Zeugswetter Andreas DCP SD , - Csaba Nagy , - Martijn van Oosterhout , Dann Corbit , - Simon Riggs , - Rick Gigger , - Christopher Kings-Lynne , - "Jim C. Nasby" , josh@agliodbs.com, - pgsql-hackers@postgresql.org, Jaime Casanova , - Peter Eisentraut -Subject: Re: [HACKERS] MERGE vs REPLACE -In-Reply-To: <200511181520.jAIFKpW02114@candle.pha.pa.us> -References: <200511181520.jAIFKpW02114@candle.pha.pa.us> -Comments: In-reply-to Bruce Momjian - message dated "Fri, 18 Nov 2005 10:20:51 -0500" -Date: Fri, 18 Nov 2005 10:34:35 -0500 -Message-ID: <6410.1132328075@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.005 required=5 tests=[AWL=0.005] -X-Spam-Score: 0.005 -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: ORr - -Bruce Momjian writes: -> Oh, good point. I was thinking just about concurrent MERGEs. However, -> it is more complicated than that. By definitaion you can not see -> changes from other transactions while your statement is being run (even -> if you increment CommandCounter), so to be atomic, you would still see -> the row even though some other transaction had deleted it. - -We would have to use the same semantics we use now for read-committed -UPDATE, that is look at the latest version of the row even though this -would not normally be visible to the transaction's snapshot. - -In the case of a serializable transaction, no doubt we should fail if -any concurrent change actually happens. - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 9: In versions below 8.0, the planner will ignore your desire to - choose an index scan if your joining column's datatypes do not - match - -From pgsql-hackers-owner+M76315@postgresql.org Sun Nov 20 12:35:46 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jAKHZjO09346 - for ; Sun, 20 Nov 2005 12:35:46 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 463E1C4B337; - Sun, 20 Nov 2005 17:35:44 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 63416D7E05 - for ; Sun, 20 Nov 2005 13:27:09 -0400 (AST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 12345-02 - for ; - Sun, 20 Nov 2005 17:27:07 +0000 (GMT) -X-Greylist: from auto-whitelisted by SQLgrey- -Received: from candle.pha.pa.us (candle.pha.pa.us [64.139.89.126]) - by svr1.postgresql.org (Postfix) with ESMTP id 264B4D7E3B - for ; Sun, 20 Nov 2005 13:27:05 -0400 (AST) -Received: (from pgman@localhost) - by candle.pha.pa.us (8.11.6/8.11.6) id jAKHQI007897; - Sun, 20 Nov 2005 12:26:18 -0500 (EST) -From: Bruce Momjian -Message-ID: <200511201726.jAKHQI007897@candle.pha.pa.us> -Subject: Re: [HACKERS] MERGE vs REPLACE -In-Reply-To: <6410.1132328075@sss.pgh.pa.us> -To: Tom Lane -Date: Sun, 20 Nov 2005 12:26:18 -0500 (EST) -cc: Dennis Bjorklund , - Zeugswetter Andreas DCP SD , - Csaba Nagy , - Martijn van Oosterhout , Dann Corbit , - Simon Riggs , - Rick Gigger , - Christopher Kings-Lynne , - "Jim C. Nasby" , josh@agliodbs.com, - pgsql-hackers@postgresql.org, Jaime Casanova , - Peter Eisentraut -X-Mailer: ELM [version 2.4ME+ PL121 (25)] -MIME-Version: 1.0 -Content-Transfer-Encoding: 7bit -Content-Type: text/plain; charset=US-ASCII -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.034 required=5 tests=[AWL=0.034] -X-Spam-Score: 0.034 -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -Tom Lane wrote: -> Bruce Momjian writes: -> > Oh, good point. I was thinking just about concurrent MERGEs. However, -> > it is more complicated than that. By definitaion you can not see -> > changes from other transactions while your statement is being run (even -> > if you increment CommandCounter), so to be atomic, you would still see -> > the row even though some other transaction had deleted it. -> -> We would have to use the same semantics we use now for read-committed -> UPDATE, that is look at the latest version of the row even though this -> would not normally be visible to the transaction's snapshot. -> -> In the case of a serializable transaction, no doubt we should fail if -> any concurrent change actually happens. - -I have some psaudocode to explain what we want for this feature, -whatever syntax we choose: - - Start - Check unique index - Found - lock row for update - if zero rows, return to start - if more than one row, fail - update row - Notfound - create savepoint - insert row into heap - lock index page - if conflicting index entry, abort savepoint, return to start - add index entry - unlock index page - -While the "notfound" case might look strange, we actually use this exact -method for inserts now, see ExecInsert() and _bt_doinsert(). -Particularly see this comment in the second function: - - /* - * If we're not allowing duplicates, make sure the key isn't already in - * the index. - * - * NOTE: obviously,_bt_check_unique can only detect keys that are already in - * the index; so it cannot defend against concurrent insertions of the - * same key. We protect against that by means of holding a write lock on - * the target page. Any other would-be inserter of the same key must - * acquire a write lock on the same target page, so only one would-be - * inserter can be making the check at one time. Furthermore, once we are - * past the check we hold write locks continuously until we have performed - * our insertion, so no later inserter can fail to see our insertion. - * (This requires some care in _bt_insertonpg.) - * - * If we must wait for another xact, we release the lock while waiting, and - * then must start over completely. - */ - -Here is the unique check error from _bt_check_unique(): - - ereport(ERROR, - (errcode(ERRCODE_UNIQUE_VIOLATION), - errmsg("duplicate key violates unique constraint \"%s\"", - RelationGetRelationName(rel)))); - -I think the problem here is that it is going to longjump() back to -postgres.c (and out of your code loop). While we have savepoints, I -think they only work coming from client applications, rather than inside -our code. Ideally you would like to be able to say: - - savepoint(); - func(); - rollback_to_savepoint(); - -but you can't, so I think you are going to have to factor out that -unique error callback and return a failure code to the caller. I -suppose some boolean flag need to be added to _bt_doinsert(), but that -is called via a function pointer for the index type, so you are going to -have to update the insert function signatures for all access methods. -The good news is that only btree supports unique indexes, according to -the documentation ("Only B-tree currently supports unique indexes") so -for the other access methods the extra parameter is just ignored. - -Another issue is multiple unique indexes. What if the first unique -index matches one row, but a different row matches the second unique -indexed column? Fail because unique checks do not identify exactly one -row? - -Or the _new_ value for the second indexed column conflicts with the -second unique index. The MERGE/REPLACE should fail. The UPDATE block -will handle this on its own, but the INSERT block will need to check for -that an really error out, rather than return to the caller, so the loop -in ExecInsertIndexTuples() has to restart on unique failure _only_ on -the first index check, not the subsequent ones. - -One simplification would be to allow MERGE/REPLACE only on a table that -has a single unique index. - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 359-1001 - + If your life is a hard drive, | 13 Roberts Road - + Christ can be your backup. | Newtown Square, Pennsylvania 19073 - ----------------------------(end of broadcast)--------------------------- -TIP 9: In versions below 8.0, the planner will ignore your desire to - choose an index scan if your joining column's datatypes do not - match - -From pgsql-hackers-owner+M76369@postgresql.org Mon Nov 21 16:49:26 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jALLnPO11702 - for ; Mon, 21 Nov 2005 16:49:25 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id E0EF1C4BBF9; - Mon, 21 Nov 2005 21:49:23 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 788FADA584 - for ; Mon, 21 Nov 2005 17:44:39 -0400 (AST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 70913-06 - for ; - Mon, 21 Nov 2005 21:44:40 +0000 (GMT) -X-Greylist: from auto-whitelisted by SQLgrey- -Received: from flake.decibel.org (flake.decibel.org [67.100.216.10]) - by svr1.postgresql.org (Postfix) with ESMTP id 128D5DA48D - for ; Mon, 21 Nov 2005 17:44:35 -0400 (AST) -Received: by flake.decibel.org (Postfix, from userid 1001) - id AD10B1525F; Mon, 21 Nov 2005 15:44:36 -0600 (CST) -Date: Mon, 21 Nov 2005 15:44:36 -0600 -From: "Jim C. Nasby" -To: Bruce Momjian -cc: mark@mark.mielke.cc, Tom Lane , - Csaba Nagy , - Martijn van Oosterhout , - Zeugswetter Andreas DCP SD , - Dann Corbit , Simon Riggs , - Rick Gigger , - Christopher Kings-Lynne , josh@agliodbs.com, - pgsql-hackers@postgresql.org, Jaime Casanova , - Peter Eisentraut -Subject: Re: [HACKERS] MERGE vs REPLACE -Message-ID: <20051121214436.GL19279@pervasive.com> -References: <20051117173117.GA27563@mark.mielke.cc> <200511180230.jAI2Uh406548@candle.pha.pa.us> -MIME-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -In-Reply-To: <200511180230.jAI2Uh406548@candle.pha.pa.us> -X-Operating-System: FreeBSD 4.11-RELEASE-p10 i386 -X-Distributed: Join the Effort! http://www.distributed.net -User-Agent: Mutt/1.5.9i -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.019 required=5 tests=[AWL=0.019] -X-Spam-Score: 0.019 -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -On Thu, Nov 17, 2005 at 09:30:43PM -0500, Bruce Momjian wrote: -> > Is the requirement for predicate locking, over and above a unique -> > constraint on an index that involves the record key, to deal with -> > the scenario of two inserts executing at the same time, both before -> > commit? -> -> No. If you have a primary key you can easily prevent duplicates. You -> need a table lock or predicate locking to prevent duplicates if you do -> not have a primary key. - -AFAIK you can also accomplish this without a table lock as long as you -have a unique index on the right set of fields and those fields are also -NOT NULL. ISTM it would be good to support that case as well, since you -might want to MERGE based on something other than the PK. --- -Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com -Pervasive Software http://pervasive.com work: 512-231-6117 -vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461 - ----------------------------(end of broadcast)--------------------------- -TIP 5: don't forget to increase your free space map settings - -From decibel@decibel.org Mon Nov 21 16:46:33 2005 -Return-path: -Received: from flake.decibel.org (flake.decibel.org [67.100.216.10]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jALLkWO11195 - for ; Mon, 21 Nov 2005 16:46:32 -0500 (EST) -Received: by flake.decibel.org (Postfix, from userid 1001) - id 78F981525F; Mon, 21 Nov 2005 15:46:30 -0600 (CST) -Date: Mon, 21 Nov 2005 15:46:30 -0600 -From: "Jim C. Nasby" -To: Tom Lane , Csaba Nagy , - Martijn van Oosterhout , - Zeugswetter Andreas DCP SD , - Dann Corbit , Simon Riggs , - Bruce Momjian , - Rick Gigger , - Christopher Kings-Lynne , josh@agliodbs.com, - pgsql-hackers@postgresql.org, Jaime Casanova , - Peter Eisentraut -Subject: Re: [HACKERS] MERGE vs REPLACE -Message-ID: <20051121214630.GM19279@pervasive.com> -References: <1132228373.10890.313.camel@coppola.muc.ecircle.de> <20051117123250.GC22933@svana.org> <1132231474.10890.317.camel@coppola.muc.ecircle.de> <25849.1132238470@sss.pgh.pa.us> <20051117151530.GU6026@ns.snowman.net> -MIME-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -In-Reply-To: <20051117151530.GU6026@ns.snowman.net> -X-Operating-System: FreeBSD 4.11-RELEASE-p10 i386 -X-Distributed: Join the Effort! http://www.distributed.net -User-Agent: Mutt/1.5.9i -Status: OR - -On Thu, Nov 17, 2005 at 10:15:30AM -0500, Stephen Frost wrote: -> I don't think MERGE can really be made to be both though, in which case -> it should really be the SQL2003 MERGE and we can make REPLACE/INSERT ON -> DUPLICATE UPDATE something else. Perhaps a special form of MERGE where -> you know it's going to be doing that locking. I really don't like the -> idea of making the SQL2003 version of MERGE be the MERGE special case -> (by requiring someone to take a table lock ahead of time or do something -> else odd). - -Anyone know off-hand what the big 3 do? If the industry consensus is -that merge should actually be REPLACE/INSERT ON DUPLICATE UPDATE then -it's probably better to follow that lead. --- -Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com -Pervasive Software http://pervasive.com work: 512-231-6117 -vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461 - -From pgsql-hackers-owner+M76416@postgresql.org Mon Nov 21 22:20:45 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jAM3KiO01066 - for ; Mon, 21 Nov 2005 22:20:45 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 16D40C4BC0E; - Tue, 22 Nov 2005 03:20:42 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 19544D6E94 - for ; Mon, 21 Nov 2005 23:20:13 -0400 (AST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 65279-06 - for ; - Tue, 22 Nov 2005 03:20:16 +0000 (GMT) -X-Greylist: from auto-whitelisted by SQLgrey- -Received: from gatekeeper.pjmodos.bounceme.net (5.176.broadband4.iol.cz [85.71.176.5]) - by svr1.postgresql.org (Postfix) with ESMTP id 63160DA4D8 - for ; Mon, 21 Nov 2005 23:20:10 -0400 (AST) -Received: from modos ([10.12.0.96] ident=PJMODOS) - by gatekeeper.pjmodos.bounceme.net with esmtp (Exim 4.50) - id 1EeOhc-0004vs-1V; Tue, 22 Nov 2005 04:20:16 +0100 -Message-ID: <43828E6C.60406@seznam.cz> -Date: Tue, 22 Nov 2005 04:20:12 +0100 -From: Petr Jelinek -User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206) -X-Accept-Language: cs, en, en-us -MIME-Version: 1.0 -To: "Jim C. Nasby" -cc: PostgreSQL-development -Subject: Re: [HACKERS] MERGE vs REPLACE -References: <1132228373.10890.313.camel@coppola.muc.ecircle.de> <20051117123250.GC22933@svana.org> <1132231474.10890.317.camel@coppola.muc.ecircle.de> <25849.1132238470@sss.pgh.pa.us> <20051117151530.GU6026@ns.snowman.net> <20051121214630.GM19279@pervasive.com> -In-Reply-To: <20051121214630.GM19279@pervasive.com> -Content-Type: text/plain; charset=ISO-8859-1; format=flowed -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by amavisd-new at hub.org -X-Amavis-Alert: BAD HEADER Improper use of control character (char 0D hex) in message header 'Received': Received: ...keeper.pjmodos.bounceme.net\r with esmtp (Ex... -X-Spam-Status: No, score=1.44 required=5 tests=[DNS_FROM_RFC_POST=1.44] -X-Spam-Score: 1.44 -X-Spam-Level: * -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -Jim C. Nasby wrote: -> On Thu, Nov 17, 2005 at 10:15:30AM -0500, Stephen Frost wrote: -> ->>I don't think MERGE can really be made to be both though, in which case ->>it should really be the SQL2003 MERGE and we can make REPLACE/INSERT ON ->>DUPLICATE UPDATE something else. Perhaps a special form of MERGE where ->>you know it's going to be doing that locking. I really don't like the ->>idea of making the SQL2003 version of MERGE be the MERGE special case ->>(by requiring someone to take a table lock ahead of time or do something ->>else odd). -> -> -> Anyone know off-hand what the big 3 do? If the industry consensus is -> that merge should actually be REPLACE/INSERT ON DUPLICATE UPDATE then -> it's probably better to follow that lead. - -It was already said here that oracle and db2 both use MERGE, dunno about -mssql. - -And yes merge CAN be used to do REPLACE (oracle uses their dummy table -for this, we can use the fact that FROM clause isn't required in postgres). - --- -Regards -Petr Jelinek (PJMODOS) - ----------------------------(end of broadcast)--------------------------- -TIP 5: don't forget to increase your free space map settings - -From pgsql-hackers-owner+M76436@postgresql.org Tue Nov 22 05:58:26 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jAMAwPO08019 - for ; Tue, 22 Nov 2005 05:58:25 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 73C27C4A48E; - Tue, 22 Nov 2005 10:58:21 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 0E86EDAFA6 - for ; Tue, 22 Nov 2005 06:57:56 -0400 (AST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 82600-10 - for ; - Tue, 22 Nov 2005 10:57:57 +0000 (GMT) -X-Greylist: from auto-whitelisted by SQLgrey- -Received: from svana.org (svana.org [203.20.62.76]) - by svr1.postgresql.org (Postfix) with ESMTP id EE3F8DAFA1 - for ; Tue, 22 Nov 2005 06:57:51 -0400 (AST) -Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian)) - id 1EeVqO-0003kj-00; Tue, 22 Nov 2005 21:57:48 +1100 -Date: Tue, 22 Nov 2005 11:57:48 +0100 -From: Martijn van Oosterhout -To: Petr Jelinek -cc: "Jim C. Nasby" , - PostgreSQL-development -Subject: Re: [HACKERS] MERGE vs REPLACE -Message-ID: <20051122105747.GC12548@svana.org> -Reply-To: Martijn van Oosterhout -References: <1132228373.10890.313.camel@coppola.muc.ecircle.de> <20051117123250.GC22933@svana.org> <1132231474.10890.317.camel@coppola.muc.ecircle.de> <25849.1132238470@sss.pgh.pa.us> <20051117151530.GU6026@ns.snowman.net> <20051121214630.GM19279@pervasive.com> <43828E6C.60406@seznam.cz> -MIME-Version: 1.0 -Content-Type: multipart/signed; micalg=pgp-sha1; - protocol="application/pgp-signature"; boundary="aT9PWwzfKXlsBJM1" -Content-Disposition: inline -In-Reply-To: <43828E6C.60406@seznam.cz> -User-Agent: Mutt/1.3.28i -X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6 -X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522 48A7 6394 F08A 0DC6 7BE6 -X-PGP-Key-URL: -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0 required=5 tests=[AWL=0.000] -X-Spam-Score: 0 -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - ---aT9PWwzfKXlsBJM1 -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -Content-Transfer-Encoding: quoted-printable - -On Tue, Nov 22, 2005 at 04:20:12AM +0100, Petr Jelinek wrote: -> It was already said here that oracle and db2 both use MERGE, dunno about= -=20 -> mssql. ->=20 -> And yes merge CAN be used to do REPLACE (oracle uses their dummy table=20 -> for this, we can use the fact that FROM clause isn't required in postgres= -). - -Statements about MERGE on the web: - -http://www.dba-oracle.com/oracle_tips_rittman_merge.htm -http://databasejournal.com/features/db2/article.php/3322041 -http://certcities.com/editorial/columns/story.asp?EditorialsID=3D51 -http://publib.boulder.ibm.com/infocenter/ids9help/index.jsp?topic=3D/com.ib= -m.sqls.doc/sqls578.htm -http://www.jdixon.dotnetdevelopersjournal.com/i_want_my_sql_2005_merge_stat= -ement.htm -http://publib.boulder.ibm.com/infocenter/db2help/topic/com.ibm.db2.udb.doc/= -admin/r0010873.htm -http://expertanswercenter.techtarget.com/eac/knowledgebaseAnswer/0,295199,s= -id63_gci978700,00.html - -Not one (*not one!*) of these mentions any special handling of -duplicate keys. They even go to pains to say that any errors cause -everything to rollback. The last one is especially interesting: - -: Is there any way to capture errors from a MERGE statement? Also, is -: there any way to know how many records were inserted or updated for the -: MERGE statement like SQL%ROWCOUNT? Any assistance greatly appreciated.=20 -: -: You capture errors the same way you would if you were doing regular -: INSERT and UPDATE statements....with exception handlers. Just include a -: WHEN OTHERS exception handler in the block where your MERGE statement -: is and have to display SQLCODE and SQLERRM if an error occurs. Then you -: can figure out which specific errors are occurring and create -: individual exception handlers for those.=20 - -There are even places that tell you how to decompose your MERGE into an -INSERT plus UPDATE statement. The real advantage of MERGE is that the -semantics prevent your updating a row you just inserted, which is -harder in the general case but easy if the executor is handling the -rows one at a time. - -Rather than trying to make MERGE do something it wasn't designed for, -we should probably be spending our efforts on triggers for error -conditions. Maybe something like: - -CREATE TRIGGER foo AFTER ERROR ON bar EXECUTE baz(); - -Where baz would be passed NEW and OLD just like a normal trigger and if -the trigger return NULL, the update is ignored. In the meantime the -function can divert the insert to another table if it likes. This seems -like a much more workable and useful addition. - -Have a nice day, ---=20 -Martijn van Oosterhout http://svana.org/kleptog/ -> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a -> tool for doing 5% of the work and then sitting around waiting for someone -> else to do the other 95% so you can sue them. - ---aT9PWwzfKXlsBJM1 -Content-Type: application/pgp-signature -Content-Disposition: inline - ------BEGIN PGP SIGNATURE----- -Version: GnuPG v1.0.6 (GNU/Linux) -Comment: For info see http://www.gnupg.org - -iD8DBQFDgvmqIB7bNG8LQkwRAr0eAKCBybl6qs2NN6kXG15tiGg+nObK0QCeP+2S -+S9F/7PZ70V8CQmZqqMn6sE= -=QsDZ ------END PGP SIGNATURE----- - ---aT9PWwzfKXlsBJM1-- - -From pgsql-hackers-owner+M76466@postgresql.org Tue Nov 22 15:31:18 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jAMKVHO11734 - for ; Tue, 22 Nov 2005 15:31:17 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id A1AC1C4BC25; - Tue, 22 Nov 2005 20:31:15 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 354D0DB257 - for ; Tue, 22 Nov 2005 16:30:47 -0400 (AST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 15163-07-3 - for ; - Tue, 22 Nov 2005 20:30:47 +0000 (GMT) -X-Greylist: from auto-whitelisted by SQLgrey- -Received: from flake.decibel.org (flake.decibel.org [67.100.216.10]) - by svr1.postgresql.org (Postfix) with ESMTP id 3E9B9DBA86 - for ; Tue, 22 Nov 2005 16:30:43 -0400 (AST) -Received: by flake.decibel.org (Postfix, from userid 1001) - id 781A61525F; Tue, 22 Nov 2005 14:30:45 -0600 (CST) -Date: Tue, 22 Nov 2005 14:30:45 -0600 -From: "Jim C. Nasby" -To: Martijn van Oosterhout -cc: Petr Jelinek , - PostgreSQL-development -Subject: Re: [HACKERS] MERGE vs REPLACE -Message-ID: <20051122203045.GC99429@pervasive.com> -References: <1132228373.10890.313.camel@coppola.muc.ecircle.de> <20051117123250.GC22933@svana.org> <1132231474.10890.317.camel@coppola.muc.ecircle.de> <25849.1132238470@sss.pgh.pa.us> <20051117151530.GU6026@ns.snowman.net> <20051121214630.GM19279@pervasive.com> <43828E6C.60406@seznam.cz> <20051122105747.GC12548@svana.org> -MIME-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -In-Reply-To: <20051122105747.GC12548@svana.org> -X-Operating-System: FreeBSD 4.11-RELEASE-p10 i386 -X-Distributed: Join the Effort! http://www.distributed.net -User-Agent: Mutt/1.5.9i -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.012 required=5 tests=[AWL=0.012] -X-Spam-Score: 0.012 -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -On Tue, Nov 22, 2005 at 11:57:48AM +0100, Martijn van Oosterhout wrote: - - -> Rather than trying to make MERGE do something it wasn't designed for, -> we should probably be spending our efforts on triggers for error -> conditions. Maybe something like: -> -> CREATE TRIGGER foo AFTER ERROR ON bar EXECUTE baz(); -> -> Where baz would be passed NEW and OLD just like a normal trigger and if -> the trigger return NULL, the update is ignored. In the meantime the -> function can divert the insert to another table if it likes. This seems -> like a much more workable and useful addition. - -I agree that we shouldn't try and distort MERGE into something fancy. -The AFTER ERROR trigger is a very interesting idea, since it could -handle many different cases. But I'm worried that people might not want -that behavior on by default for everything done against some table. I -think it'd be better to have some way to specify in a command that -you want to use some kind of error-handling trigger. Though presumably -the underlying framework would be same, so it shouldn't be hard to -support both. --- -Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com -Pervasive Software http://pervasive.com work: 512-231-6117 -vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461 - ----------------------------(end of broadcast)--------------------------- -TIP 4: Have you searched our list archives? - - http://archives.postgresql.org - -From pgsql-hackers-owner+M76499@postgresql.org Wed Nov 23 04:10:09 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jAN9A8O01978 - for ; Wed, 23 Nov 2005 04:10:09 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 8270EC4BC25; - Wed, 23 Nov 2005 09:10:04 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 39862DB27A - for ; Wed, 23 Nov 2005 05:09:40 -0400 (AST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 89452-02 - for ; - Wed, 23 Nov 2005 09:09:40 +0000 (GMT) -X-Greylist: from auto-whitelisted by SQLgrey- -Received: from svana.org (svana.org [203.20.62.76]) - by svr1.postgresql.org (Postfix) with ESMTP id C6404DB088 - for ; Wed, 23 Nov 2005 05:09:36 -0400 (AST) -Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian)) - id 1Eeqd7-0002FD-00; Wed, 23 Nov 2005 20:09:29 +1100 -Date: Wed, 23 Nov 2005 10:09:29 +0100 -From: Martijn van Oosterhout -To: Petr Jelinek -cc: Jaime Casanova , - PostgreSQL-development , - "Jim C. Nasby" -Subject: Re: [HACKERS] MERGE vs REPLACE -Message-ID: <20051123090926.GA8374@svana.org> -Reply-To: Martijn van Oosterhout -References: <1132228373.10890.313.camel@coppola.muc.ecircle.de> <20051117123250.GC22933@svana.org> <1132231474.10890.317.camel@coppola.muc.ecircle.de> <25849.1132238470@sss.pgh.pa.us> <20051117151530.GU6026@ns.snowman.net> <20051121214630.GM19279@pervasive.com> <43828E6C.60406@seznam.cz> <4383A8A8.30805@seznam.cz> -MIME-Version: 1.0 -Content-Type: multipart/signed; micalg=pgp-sha1; - protocol="application/pgp-signature"; boundary="fdj2RfSjLxBAspz7" -Content-Disposition: inline -In-Reply-To: <4383A8A8.30805@seznam.cz> -User-Agent: Mutt/1.3.28i -X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6 -X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522 48A7 6394 F08A 0DC6 7BE6 -X-PGP-Key-URL: -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0 required=5 tests=[AWL=0.000] -X-Spam-Score: 0 -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - ---fdj2RfSjLxBAspz7 -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -Content-Transfer-Encoding: quoted-printable - -On Wed, Nov 23, 2005 at 12:24:24AM +0100, Petr Jelinek wrote: -> Btw about that keys, oracle gives error on many-to-one or many-to-many=20 -> relationship between the source and target tables. - -The standard has something called a "cardinality violation" if the -to-be-merged table doesn't match 1-1 with the rest of the statement. If -I had access to an Oracle I'd run two tests on MERGE: - -1. Does the joining column have to have an index? For example, make a -column that's full of unique values but no unique index. According to -my reading of the the standard, this should still work (just slower). - -2. Additionally, only the rows involved in the MERGE need to be -uniquely referenced, so if you add duplicate values but add a WHERE -clause to exclude those, it should also work. - -My feeling is that requiring an index will limit it's usefulness as a -general tool. - -Have a nice day, ---=20 -Martijn van Oosterhout http://svana.org/kleptog/ -> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a -> tool for doing 5% of the work and then sitting around waiting for someone -> else to do the other 95% so you can sue them. - ---fdj2RfSjLxBAspz7 -Content-Type: application/pgp-signature -Content-Disposition: inline - ------BEGIN PGP SIGNATURE----- -Version: GnuPG v1.0.6 (GNU/Linux) -Comment: For info see http://www.gnupg.org - -iD8DBQFDhDHFIB7bNG8LQkwRAq+3AJ4kprR3WbWOegJGf9JTVsNLa11/jQCfTmH8 -6o2CsvPgg1g8uOnjbHoGMHo= -=D4Cr ------END PGP SIGNATURE----- - ---fdj2RfSjLxBAspz7-- - -From pgsql-hackers-owner+M76510@postgresql.org Wed Nov 23 14:25:05 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jANJP4O22462 - for ; Wed, 23 Nov 2005 14:25:04 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 3EF3EC4BBCD; - Wed, 23 Nov 2005 19:25:02 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 94C5FDB839 - for ; Wed, 23 Nov 2005 15:24:39 -0400 (AST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 81352-06 - for ; - Wed, 23 Nov 2005 19:24:39 +0000 (GMT) -X-Greylist: delayed 00:31:59.455806 by SQLgrey- -Received: from mecho.sysmaster.com (unknown [65.119.37.45]) - by svr1.postgresql.org (Postfix) with ESMTP id 5078FDA7A5 - for ; Wed, 23 Nov 2005 15:24:36 -0400 (AST) -Received: from [127.0.0.1] (office.sysmaster.com [65.113.143.60]) - by mecho.sysmaster.com (8.11.6/8.11.6) with ESMTP id jANIqMB22433; - Wed, 23 Nov 2005 10:52:22 -0800 -Message-ID: <4384BA5F.4080402@sysmaster.com> -Date: Wed, 23 Nov 2005 10:52:15 -0800 -From: Lyubomir Petrov -User-Agent: Thunderbird 1.5 (Windows/20051025) -MIME-Version: 1.0 -To: Martijn van Oosterhout -cc: PostgreSQL-development -Subject: Re: [HACKERS] MERGE vs REPLACE -References: <1132228373.10890.313.camel@coppola.muc.ecircle.de> <20051117123250.GC22933@svana.org> <1132231474.10890.317.camel@coppola.muc.ecircle.de> <25849.1132238470@sss.pgh.pa.us> <20051117151530.GU6026@ns.snowman.net> <20051121214630.GM19279@pervasive.com> <43828E6C.60406@seznam.cz> <4383A8A8.30805@seznam.cz> <20051123090926.GA8374@svana.org> -In-Reply-To: <20051123090926.GA8374@svana.org> -Content-Type: text/plain; charset=ISO-8859-1; format=flowed -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0 required=5 tests=[none] -X-Spam-Score: 0 -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -Martijn, - -Here is a quick test (Oracle 10.1.0.3/Linux): - - -SQL> select banner from v$version; -BANNER ----------------------------------------------------------------- -Oracle Database 10g Enterprise Edition Release 10.1.0.3.0 - Prod -PL/SQL Release 10.1.0.3.0 - Production -CORE 10.1.0.3.0 Production -TNS for Linux: Version 10.1.0.3.0 - Production -NLSRTL Version 10.1.0.3.0 - Production - -SQL> select * from merge_test_1; - ID NAME ----------- -------------------- - 1 aaa - 2 bbb - 3 ccc - 4 ddd - 5 eee - 1 xxx -6 rows selected. - -SQL> select * from merge_test_2; - ID NAME ----------- -------------------- - 1 AAA - 2 BBB - 6 FFF - -SQL> select index_name from user_indexes where table_name like -'merge_test%'; -no rows selected - -SQL> merge into merge_test_1 a1 - 2 using merge_test_2 a2 - 3 on (a1.id = a2.id) - 4 when matched then - 5 update set a1.name = a2.name - 6 when not matched then - 7 insert (id, name) values (a2.id, a2.name); -4 rows merged. - -SQL> select * from merge_test_1; - ID NAME ----------- -------------------- - 1 AAA - 2 BBB - 3 ccc - 4 ddd - 5 eee - 1 AAA - 6 FFF -7 rows selected. - - - -Regards, -Lubomir Petrov - - - -Martijn van Oosterhout wrote: -> On Wed, Nov 23, 2005 at 12:24:24AM +0100, Petr Jelinek wrote: -> ->> Btw about that keys, oracle gives error on many-to-one or many-to-many ->> relationship between the source and target tables. ->> -> -> The standard has something called a "cardinality violation" if the -> to-be-merged table doesn't match 1-1 with the rest of the statement. If -> I had access to an Oracle I'd run two tests on MERGE: -> -> 1. Does the joining column have to have an index? For example, make a -> column that's full of unique values but no unique index. According to -> my reading of the the standard, this should still work (just slower). -> -> 2. Additionally, only the rows involved in the MERGE need to be -> uniquely referenced, so if you add duplicate values but add a WHERE -> clause to exclude those, it should also work. -> -> My feeling is that requiring an index will limit it's usefulness as a -> general tool. -> -> Have a nice day, -> - - ----------------------------(end of broadcast)--------------------------- -TIP 3: Have you checked our extensive FAQ? - - http://www.postgresql.org/docs/faq - -From pgsql-hackers-owner+M76513@postgresql.org Wed Nov 23 17:23:34 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jANMNXO20323 - for ; Wed, 23 Nov 2005 17:23:33 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id C020EC4A48E; - Wed, 23 Nov 2005 22:23:30 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 03F71DBAA3 - for ; Wed, 23 Nov 2005 18:23:06 -0400 (AST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 92598-01 - for ; - Wed, 23 Nov 2005 22:23:06 +0000 (GMT) -X-Greylist: from auto-whitelisted by SQLgrey- -Received: from smtp018.mail.yahoo.com (smtp018.mail.yahoo.com [216.136.174.115]) - by svr1.postgresql.org (Postfix) with SMTP id 121D3DBA93 - for ; Wed, 23 Nov 2005 18:23:01 -0400 (AST) -Received: (qmail 47652 invoked from network); 23 Nov 2005 22:22:56 -0000 -DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; - s=s1024; d=Yahoo.com; - h=Received:Message-ID:Date:From:User-Agent:X-Accept-Language:MIME-Version:To:CC:Subject:References:In-Reply-To:Content-Type:Content-Transfer-Encoding; - b=ikj/Ob+3p22lsO9ikWyS1RkIbs6fq47zO2HF4jU7ZStjg5F/smOj0F+YkTeoepYDBoiKtd5pr0vxODe6iSXdUmQgXKe4BvzaypEm8KFqgR2kKtlpW/QAy1arUUnD1DoFpx/SJgPFt1V4S//lgGTpWw5v99EeoYfapzNmNV0hgQI= ; -Received: from unknown (HELO jupiter.black-lion.info) (janwieck@68.80.245.191 with login) - by smtp018.mail.yahoo.com with SMTP; 23 Nov 2005 22:22:55 -0000 -Received: from [172.21.8.23] (mars.black-lion.info [192.168.192.101]) - (authenticated bits=0) - by jupiter.black-lion.info (8.12.10/8.12.9) with ESMTP id jANLtT1o054955; - Wed, 23 Nov 2005 16:55:29 -0500 (EST) - (envelope-from JanWieck@Yahoo.com) -Message-ID: <4384E54D.8000500@Yahoo.com> -Date: Wed, 23 Nov 2005 16:55:25 -0500 -From: Jan Wieck -User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.1) Gecko/20040707 -X-Accept-Language: en-us, en -MIME-Version: 1.0 -To: Tom Lane -cc: Peter Eisentraut , pgsql-hackers@postgresql.org, - Josh Berkus , Jaime Casanova -Subject: Re: [HACKERS] someone working to add merge? -References: <200511111924.41532.peter_e@gmx.net> <20561.1131734697@sss.pgh.pa.us> <200511112004.35876.peter_e@gmx.net> <20899.1131736841@sss.pgh.pa.us> -In-Reply-To: <20899.1131736841@sss.pgh.pa.us> -Content-Type: text/plain; charset=us-ascii; format=flowed -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.359 required=5 tests=[AWL=-0.120, - DNS_FROM_RFC_ABUSE=0.479] -X-Spam-Score: 0.359 -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -On 11/11/2005 2:20 PM, Tom Lane wrote: - -> Peter Eisentraut writes: ->> Tom Lane wrote: ->>> Surely they require a unique constraint --- else the behavior isn't ->>> even well defined, is it? -> ->> They require that the merge condition does not match for more than one ->> row, but since the merge condition can do just about anything, there is ->> no guarantee that a unique constraint encompasses it. -> -> ISTM to be a reasonable implementation restriction that there be a -> constraint by which the system can prove that there is at most one -> matching row. Per other comments in this thread, we'd not be the only -> implementation making such a restriction. -> -> (Certainly, if I were a DBA and were told that the performance of MERGE -> would go to hell in a handbasket if I had no such constraint, I'd make -> sure there was one. I don't think there is very much of a use-case for -> the general scenario.) - -Such restriction does look reasonable. Especially because ... - -The largest problem I see with MERGE is the question of BEFORE triggers. -Consider a BEFORE INSERT trigger that modifies a third table, after -which the constraint or whatever post-heap_insert-attempt we might use -detects a conflict. How do we undo the actions of the BEFORE trigger? -The only way to do that is to plan the query as a nestloop, with the -USING part as the outer loop. If the (updating) scan of the INTO -relation did not hit any tuple, then do the INSERT. We can only undo the -side effects of any BEFORE trigger by wrapping each and evey nested INTO -relation insert attempt into its own subtransaction. - -Sure, we "could" of course do the insert and then rescan the whole thing -with read-committed to see if our row is now the only one ... needless -to say that in the case of a sequential scan inside the loop, that -nestloop will suck big times even without that second scan. But ... hmmm -... we could get away with that and if we don't find a constraint that -will ensure uniqueness, then we do a rescan to check for it. But I would -vote for a "please_no_notice_about_stupid_usage_of_merge" runtime option -that suppresses the corresponding NOTICE. - - -Jan - --- -#======================================================================# -# It's easier to get forgiveness for being wrong than for being right. # -# Let's break this rule - forgive me. # -#================================================== JanWieck@Yahoo.com # - ----------------------------(end of broadcast)--------------------------- -TIP 4: Have you searched our list archives? - - http://archives.postgresql.org - -From pgsql-hackers-owner+M76524@postgresql.org Thu Nov 24 01:31:30 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jAO6VTO16396 - for ; Thu, 24 Nov 2005 01:31:30 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id D9160C4A48E; - Thu, 24 Nov 2005 06:31:25 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 42FE3DBC43 - for ; Thu, 24 Nov 2005 02:31:01 -0400 (AST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 56362-10 - for ; - Thu, 24 Nov 2005 06:31:00 +0000 (GMT) -X-Greylist: from auto-whitelisted by SQLgrey- -Received: from svana.org (svana.org [203.20.62.76]) - by svr1.postgresql.org (Postfix) with ESMTP id A1169DBC30 - for ; Thu, 24 Nov 2005 02:30:58 -0400 (AST) -Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian)) - id 1EfAcw-0004xu-00; Thu, 24 Nov 2005 17:30:38 +1100 -Date: Thu, 24 Nov 2005 07:30:37 +0100 -From: Martijn van Oosterhout -To: Jan Wieck -cc: Tom Lane , Peter Eisentraut , - pgsql-hackers@postgresql.org, Josh Berkus , - Jaime Casanova -Subject: Re: [HACKERS] someone working to add merge? -Message-ID: <20051124063034.GA18750@svana.org> -Reply-To: Martijn van Oosterhout -References: <200511111924.41532.peter_e@gmx.net> <20561.1131734697@sss.pgh.pa.us> <200511112004.35876.peter_e@gmx.net> <20899.1131736841@sss.pgh.pa.us> <4384E54D.8000500@Yahoo.com> -MIME-Version: 1.0 -Content-Type: multipart/signed; micalg=pgp-sha1; - protocol="application/pgp-signature"; boundary="45Z9DzgjV8m4Oswq" -Content-Disposition: inline -In-Reply-To: <4384E54D.8000500@Yahoo.com> -User-Agent: Mutt/1.3.28i -X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6 -X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522 48A7 6394 F08A 0DC6 7BE6 -X-PGP-Key-URL: -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0 required=5 tests=[AWL=0.000] -X-Spam-Score: 0 -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - ---45Z9DzgjV8m4Oswq -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -Content-Transfer-Encoding: quoted-printable - -On Wed, Nov 23, 2005 at 04:55:25PM -0500, Jan Wieck wrote: -> The largest problem I see with MERGE is the question of BEFORE triggers.= -=20 -> Consider a BEFORE INSERT trigger that modifies a third table, after=20 -> which the constraint or whatever post-heap_insert-attempt we might use=20 -> detects a conflict. How do we undo the actions of the BEFORE trigger?=20 -> The only way to do that is to plan the query as a nestloop, with the=20 -> USING part as the outer loop. If the (updating) scan of the INTO=20 -> relation did not hit any tuple, then do the INSERT. We can only undo the= -=20 -> side effects of any BEFORE trigger by wrapping each and evey nested INTO= -=20 -> relation insert attempt into its own subtransaction. - -Umm, if there are any errors you abort the transaction, just like any -other case. ACID requires that either the whole statement is done, or -none. If a trigger causes the INSERT or UPDATE to fail you have no -choice but to abort the transaction. - -Besides, someone posted an example on Oracle, they don't require an -index so I don't think we realistically can say that people need one. -If two concurrent MERGEs, which can't see eachothers output, both end -up INSERTing, that not an error unless the user has a UNIQUE -constraint, so the problem vanishes. - -Have a nice day, ---=20 -Martijn van Oosterhout http://svana.org/kleptog/ -> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a -> tool for doing 5% of the work and then sitting around waiting for someone -> else to do the other 95% so you can sue them. - ---45Z9DzgjV8m4Oswq -Content-Type: application/pgp-signature -Content-Disposition: inline - ------BEGIN PGP SIGNATURE----- -Version: GnuPG v1.0.6 (GNU/Linux) -Comment: For info see http://www.gnupg.org - -iD8DBQFDhV4HIB7bNG8LQkwRAgXZAKCBD3QOdBVT3GmCqEQj9tfapJq38QCeOdvW -VXixm+YDkCCWkSLEQcUtixM= -=3H5a ------END PGP SIGNATURE----- - ---45Z9DzgjV8m4Oswq-- - -From pgsql-hackers-owner+M76549@postgresql.org Thu Nov 24 11:12:14 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jAOGCDO07384 - for ; Thu, 24 Nov 2005 11:12:13 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 305F4C4A48E; - Thu, 24 Nov 2005 16:12:12 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 6A539DBC1D - for ; Thu, 24 Nov 2005 12:11:46 -0400 (AST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 60324-06-2 - for ; - Thu, 24 Nov 2005 16:11:44 +0000 (GMT) -X-Greylist: from auto-whitelisted by SQLgrey- -Received: from smtp011.mail.yahoo.com (smtp011.mail.yahoo.com [216.136.173.31]) - by svr1.postgresql.org (Postfix) with SMTP id 11B74DBBF3 - for ; Thu, 24 Nov 2005 12:11:42 -0400 (AST) -Received: (qmail 72237 invoked from network); 24 Nov 2005 16:11:41 -0000 -DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; - s=s1024; d=Yahoo.com; - h=Received:Message-ID:Date:From:User-Agent:X-Accept-Language:MIME-Version:To:CC:Subject:References:In-Reply-To:Content-Type:Content-Transfer-Encoding; - b=dgj7516dhcBQJ4gSnC2f7PmbQi7TxXH8v8ahOhOPDm/53Yj9wVKNG2jTbp+j2crVe+Tw7VA5EEhS2YqaHTmD8Fs3np6yVP4J1VN/SF7+aywZkiiRJUSEAe3qYpGIGRFxtnKr3K3M6TufdeJZuEKu4O3YCW+IZi8SCulBwKV5jfM= ; -Received: from unknown (HELO jupiter.black-lion.info) (janwieck@68.80.245.191 with login) - by smtp011.mail.yahoo.com with SMTP; 24 Nov 2005 16:11:41 -0000 -Received: from [172.21.8.23] (mars.black-lion.info [192.168.192.101]) - (authenticated bits=0) - by jupiter.black-lion.info (8.12.10/8.12.9) with ESMTP id jAOGBc1o058226; - Thu, 24 Nov 2005 11:11:38 -0500 (EST) - (envelope-from JanWieck@Yahoo.com) -Message-ID: <4385E636.4070503@Yahoo.com> -Date: Thu, 24 Nov 2005 11:11:34 -0500 -From: Jan Wieck -User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.1) Gecko/20040707 -X-Accept-Language: en-us, en -MIME-Version: 1.0 -To: Martijn van Oosterhout -cc: Tom Lane , Peter Eisentraut , - pgsql-hackers@postgresql.org, Josh Berkus , - Jaime Casanova -Subject: Re: [HACKERS] someone working to add merge? -References: <200511111924.41532.peter_e@gmx.net> <20561.1131734697@sss.pgh.pa.us> <200511112004.35876.peter_e@gmx.net> <20899.1131736841@sss.pgh.pa.us> <4384E54D.8000500@Yahoo.com> <20051124063034.GA18750@svana.org> -In-Reply-To: <20051124063034.GA18750@svana.org> -Content-Type: text/plain; charset=us-ascii; format=flowed -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.359 required=5 tests=[AWL=-0.120, - DNS_FROM_RFC_ABUSE=0.479] -X-Spam-Score: 0.359 -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -On 11/24/2005 1:30 AM, Martijn van Oosterhout wrote: - -> On Wed, Nov 23, 2005 at 04:55:25PM -0500, Jan Wieck wrote: ->> The largest problem I see with MERGE is the question of BEFORE triggers. ->> Consider a BEFORE INSERT trigger that modifies a third table, after ->> which the constraint or whatever post-heap_insert-attempt we might use ->> detects a conflict. How do we undo the actions of the BEFORE trigger? ->> The only way to do that is to plan the query as a nestloop, with the ->> USING part as the outer loop. If the (updating) scan of the INTO ->> relation did not hit any tuple, then do the INSERT. We can only undo the ->> side effects of any BEFORE trigger by wrapping each and evey nested INTO ->> relation insert attempt into its own subtransaction. -> -> Umm, if there are any errors you abort the transaction, just like any -> other case. ACID requires that either the whole statement is done, or -> none. If a trigger causes the INSERT or UPDATE to fail you have no -> choice but to abort the transaction. - -I guess you misunderstood. What I am talking about is a problem in the -order of execution. since we don't have predicate locking, there is a -possibility that our implementation of MERGE decides to do an INSERT -while another transaction does the same. What has to happen is that the -BEFORE INSERT trigger is called, then the heap tuple inserted, then the -index tuples created. At this time, the duplicate key error occurs, -telling us that we had a conflict and that we have to try an UPDATE -instead. That means, in the end this particular row's INSERT has never -happened and we have to undo the BEFORE INSERT triggers actions too. - -> -> Besides, someone posted an example on Oracle, they don't require an -> index so I don't think we realistically can say that people need one. -> If two concurrent MERGEs, which can't see eachothers output, both end -> up INSERTing, that not an error unless the user has a UNIQUE -> constraint, so the problem vanishes. - -Not following the semantics is an error. MERGE is not supposed to do -multiple inserts for the same match, concurrency or not. - - -Jan - --- -#======================================================================# -# It's easier to get forgiveness for being wrong than for being right. # -# Let's break this rule - forgive me. # -#================================================== JanWieck@Yahoo.com # - ----------------------------(end of broadcast)--------------------------- -TIP 4: Have you searched our list archives? - - http://archives.postgresql.org - -From pgsql-hackers-owner+M76581@postgresql.org Fri Nov 25 07:14:48 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jAPCElB26143 - for ; Fri, 25 Nov 2005 07:14:47 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id 46071C4BC79; - Fri, 25 Nov 2005 12:14:46 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id A32A1D72AA - for ; Fri, 25 Nov 2005 08:14:18 -0400 (AST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 86344-02 - for ; - Fri, 25 Nov 2005 12:14:19 +0000 (GMT) -X-Greylist: from auto-whitelisted by SQLgrey- -Received: from svana.org (svana.org [203.20.62.76]) - by svr1.postgresql.org (Postfix) with ESMTP id 14D00D6FFA - for ; Fri, 25 Nov 2005 08:14:14 -0400 (AST) -Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian)) - id 1EfcSp-00058u-00; Fri, 25 Nov 2005 23:14:03 +1100 -Date: Fri, 25 Nov 2005 13:14:03 +0100 -From: Martijn van Oosterhout -To: Jan Wieck -cc: Tom Lane , Peter Eisentraut , - pgsql-hackers@postgresql.org, Josh Berkus , - Jaime Casanova -Subject: Re: [HACKERS] someone working to add merge? -Message-ID: <20051125121402.GB16970@svana.org> -Reply-To: Martijn van Oosterhout -References: <200511111924.41532.peter_e@gmx.net> <20561.1131734697@sss.pgh.pa.us> <200511112004.35876.peter_e@gmx.net> <20899.1131736841@sss.pgh.pa.us> <4384E54D.8000500@Yahoo.com> <20051124063034.GA18750@svana.org> <4385E636.4070503@Yahoo.com> -MIME-Version: 1.0 -Content-Type: multipart/signed; micalg=pgp-sha1; - protocol="application/pgp-signature"; boundary="2B/JsCI69OhZNC5r" -Content-Disposition: inline -In-Reply-To: <4385E636.4070503@Yahoo.com> -User-Agent: Mutt/1.3.28i -X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6 -X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522 48A7 6394 F08A 0DC6 7BE6 -X-PGP-Key-URL: -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0 required=5 tests=[AWL=0.000] -X-Spam-Score: 0 -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - ---2B/JsCI69OhZNC5r -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -Content-Transfer-Encoding: quoted-printable - -On Thu, Nov 24, 2005 at 11:11:34AM -0500, Jan Wieck wrote: -> On 11/24/2005 1:30 AM, Martijn van Oosterhout wrote: -> >Umm, if there are any errors you abort the transaction, just like any -> >other case. ACID requires that either the whole statement is done, or -> >none. If a trigger causes the INSERT or UPDATE to fail you have no -> >choice but to abort the transaction. ->=20 -> I guess you misunderstood. What I am talking about is a problem in the=20 -> order of execution. since we don't have predicate locking, there is a=20 -> possibility that our implementation of MERGE decides to do an INSERT=20 -> while another transaction does the same. What has to happen is that the= -=20 -> BEFORE INSERT trigger is called, then the heap tuple inserted, then the= -=20 -> index tuples created. At this time, the duplicate key error occurs,=20 -> telling us that we had a conflict and that we have to try an UPDATE=20 -> instead. That means, in the end this particular row's INSERT has never=20 -> happened and we have to undo the BEFORE INSERT triggers actions too. - -But I'm not sure we're supposed to handle that case anyway. Oracle at -least doesn't require an index on the table being merged. And if I look -at it from a visibility view point, if someone else does an INSERT in -another transaction, then MERGE cannot see it and thus it will INSERT -too. This isn't an error. - -Consider the case of a deferred unique constraint (Postgres doesn't -support these yet but bear with me). Say in one session you start a -transaction, do a MERGE and then a few other statements. In the -meantime in another session someone inserts a row on the table you -merged. Are you asserting that the first session should undo -everything, do an UPDATE instead of an INSERT and redo all your queries -since? Obviously not. - -Though the above relies on something Postgres doesn't support, but you -would be able to emulate the some without a unique key. For example: - -Session 1: -CREATE TEMP TABLE foo (id integer, val integer); - -BEGIN; -SELECT * FROM foo; - -Session 2: -BEGIN; - -INSERT INTO foo (1,3); - -Session 1: -MERGE INTO foo - USING (SELECT 1) - ON (foo.id =3D 1) - WHEN MATCHED THEN UPDATE SET val =3D 1 - WHEN NOT MATCHED THEN INSERT (id,val) VALUES (1,2); - -Session 2: -COMMIT; - -Session 1: -COMMIT; - -Now, (IMO) in serializable mode, the MERGE should block on reading the -row inserted by the second session and when that commits do the UPDATE, -thus leaving you with a table with one row (1.1). - -In read committed mode, the MERGE shouldn't block and you should end up -with a table with two rows (1,3) and (1,2). - -If you switch the order of the insert and merge you should get the same -results in both cases, (1,2) and (1,3). - -I think you are arguing that the result should be (1,1) in all cases. I -honestly don't see how that is feasible and certainly not supported by -my reading of the standard. I would be interested to know how other -databases handle the above case. - -> Not following the semantics is an error. MERGE is not supposed to do=20 -> multiple inserts for the same match, concurrency or not. - -Yes, any single MERGE cannot insert twice. However, two MERGEs running -concurrently, or a MERGE with an INSERT/UPDATE/DELETE in another -session could very well end up with multiple rows on the same key. I -maintain that MERGE has no special rules w.r.t. visibility or locking -and we should not be acting as if it does. If at the end of the -transaction there a duplicate key we should throw the error and let the -client deal with it. - -Have a nice day, ---=20 -Martijn van Oosterhout http://svana.org/kleptog/ -> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a -> tool for doing 5% of the work and then sitting around waiting for someone -> else to do the other 95% so you can sue them. - ---2B/JsCI69OhZNC5r -Content-Type: application/pgp-signature -Content-Disposition: inline - ------BEGIN PGP SIGNATURE----- -Version: GnuPG v1.0.6 (GNU/Linux) -Comment: For info see http://www.gnupg.org - -iD8DBQFDhwAKIB7bNG8LQkwRAvpQAJsHNO/wnfcH0QwNQ7AKXkV5CzCz9wCfVJBH -/iCYKxdxjTOkw6+M2PgPAjY= -=ERXt ------END PGP SIGNATURE----- - ---2B/JsCI69OhZNC5r-- - -From pgsql-hackers-owner+M76584@postgresql.org Fri Nov 25 09:15:33 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jAPEFWB13482 - for ; Fri, 25 Nov 2005 09:15:32 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id AFCB6C4BC79; - Fri, 25 Nov 2005 14:15:30 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 039CCD774A - for ; Fri, 25 Nov 2005 10:14:59 -0400 (AST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 13125-10 - for ; - Fri, 25 Nov 2005 14:15:01 +0000 (GMT) -X-Greylist: from auto-whitelisted by SQLgrey- -Received: from smtp015.mail.yahoo.com (smtp015.mail.yahoo.com [216.136.173.59]) - by svr1.postgresql.org (Postfix) with SMTP id 55EA1D6822 - for ; Fri, 25 Nov 2005 10:14:55 -0400 (AST) -Received: (qmail 6658 invoked from network); 25 Nov 2005 14:14:55 -0000 -DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; - s=s1024; d=Yahoo.com; - h=Received:Message-ID:Date:From:User-Agent:X-Accept-Language:MIME-Version:To:CC:Subject:References:In-Reply-To:Content-Type:Content-Transfer-Encoding; - b=3IYwOeJgAhkmP/C6LDpv1wsSJCeqr4E0XFCc+cW2u3ZWf6NbmJ19+1NqiyPs/4RVzSH0M8FIPcRZDB1e3bkKQHt3wxdGJrfdxMi5d7DjlC59xirRpZGzUAD3yusU8lg3uE2G/TraLtC6VVpbJo8+rYerPOg+daspmuRqhlcXUjg= ; -Received: from unknown (HELO jupiter.black-lion.info) (janwieck@68.80.245.191 with login) - by smtp015.mail.yahoo.com with SMTP; 25 Nov 2005 14:14:55 -0000 -Received: from [172.21.8.23] (mars.black-lion.info [192.168.192.101]) - (authenticated bits=0) - by jupiter.black-lion.info (8.12.10/8.12.9) with ESMTP id jAPEEq1o062023; - Fri, 25 Nov 2005 09:14:52 -0500 (EST) - (envelope-from JanWieck@Yahoo.com) -Message-ID: <43871C57.6060000@Yahoo.com> -Date: Fri, 25 Nov 2005 09:14:47 -0500 -From: Jan Wieck -User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.1) Gecko/20040707 -X-Accept-Language: en-us, en -MIME-Version: 1.0 -To: Martijn van Oosterhout -cc: Tom Lane , Peter Eisentraut , - pgsql-hackers@postgresql.org, Josh Berkus , - Jaime Casanova -Subject: Re: [HACKERS] someone working to add merge? -References: <200511111924.41532.peter_e@gmx.net> <20561.1131734697@sss.pgh.pa.us> <200511112004.35876.peter_e@gmx.net> <20899.1131736841@sss.pgh.pa.us> <4384E54D.8000500@Yahoo.com> <20051124063034.GA18750@svana.org> <4385E636.4070503@Yahoo.com> <20051125121402.GB16970@svana.org> -In-Reply-To: <20051125121402.GB16970@svana.org> -Content-Type: text/plain; charset=us-ascii; format=flowed -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.359 required=5 tests=[AWL=-0.120, - DNS_FROM_RFC_ABUSE=0.479] -X-Spam-Score: 0.359 -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -On 11/25/2005 7:14 AM, Martijn van Oosterhout wrote: -> On Thu, Nov 24, 2005 at 11:11:34AM -0500, Jan Wieck wrote: ->> I guess you misunderstood. [...] -> -> But I'm not sure we're supposed to handle that case anyway. Oracle at -> least doesn't require an index on the table being merged. And if I look -> at it from a visibility view point, if someone else does an INSERT in -> another transaction, then MERGE cannot see it and thus it will INSERT -> too. This isn't an error. - -Hmmm ... so you maintain that MERGE without an explicit LOCK TABLE, done -by the user before performing the MERGE, can create duplicate rows (WRT -the merge condition) and consequently raise a duplicate key error if -there is a UNIQUE constraint. - -If that is what the standard describes, then it can be implemented -without any sort of index or constraint requirement. The query tree for -MERGE will have the INTO relation as a left outer join. In the case of a -match of this outer join, one set of targetlist expressions is used to -form the result tuple containing the INTO-relations ctid. That result -tuple us useable for heap_update() or heap_delete(). In the case of -no-match another set of target list expressions is used, suitable for -heap_insert(). This way, MERGE will work with one single sequential scan -of the INTO relation in case there is no suitable index. - - -Jan - --- -#======================================================================# -# It's easier to get forgiveness for being wrong than for being right. # -# Let's break this rule - forgive me. # -#================================================== JanWieck@Yahoo.com # - ----------------------------(end of broadcast)--------------------------- -TIP 9: In versions below 8.0, the planner will ignore your desire to - choose an index scan if your joining column's datatypes do not - match - -From pgsql-hackers-owner+M76588@postgresql.org Fri Nov 25 09:55:41 2005 -Return-path: -Received: from ams.hub.org (ams.hub.org [200.46.204.13]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jAPEtfB01898 - for ; Fri, 25 Nov 2005 09:55:41 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by ams.hub.org (Postfix) with ESMTP id D0574C4BCA2; - Fri, 25 Nov 2005 14:55:39 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (av.hub.org [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 9BABBD7693 - for ; Fri, 25 Nov 2005 10:55:13 -0400 (AST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 21958-02 - for ; - Fri, 25 Nov 2005 14:55:16 +0000 (GMT) -X-Greylist: from auto-whitelisted by SQLgrey- -Received: from svana.org (svana.org [203.20.62.76]) - by svr1.postgresql.org (Postfix) with ESMTP id C3689D7121 - for ; Fri, 25 Nov 2005 10:55:08 -0400 (AST) -Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian)) - id 1Efeyd-0005xQ-00; Sat, 26 Nov 2005 01:55:03 +1100 -Date: Fri, 25 Nov 2005 15:55:03 +0100 -From: Martijn van Oosterhout -To: Jan Wieck -cc: Tom Lane , Peter Eisentraut , - pgsql-hackers@postgresql.org, Josh Berkus , - Jaime Casanova -Subject: Re: [HACKERS] someone working to add merge? -Message-ID: <20051125145501.GD16970@svana.org> -Reply-To: Martijn van Oosterhout -References: <200511111924.41532.peter_e@gmx.net> <20561.1131734697@sss.pgh.pa.us> <200511112004.35876.peter_e@gmx.net> <20899.1131736841@sss.pgh.pa.us> <4384E54D.8000500@Yahoo.com> <20051124063034.GA18750@svana.org> <4385E636.4070503@Yahoo.com> <20051125121402.GB16970@svana.org> <43871C57.6060000@Yahoo.com> -MIME-Version: 1.0 -Content-Type: multipart/signed; micalg=pgp-sha1; - protocol="application/pgp-signature"; boundary="NklN7DEeGtkPCoo3" -Content-Disposition: inline -In-Reply-To: <43871C57.6060000@Yahoo.com> -User-Agent: Mutt/1.3.28i -X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6 -X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522 48A7 6394 F08A 0DC6 7BE6 -X-PGP-Key-URL: -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0 required=5 tests=[AWL=0.000] -X-Spam-Score: 0 -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - ---NklN7DEeGtkPCoo3 -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -Content-Transfer-Encoding: quoted-printable - -On Fri, Nov 25, 2005 at 09:14:47AM -0500, Jan Wieck wrote: -> Hmmm ... so you maintain that MERGE without an explicit LOCK TABLE, done= -=20 -> by the user before performing the MERGE, can create duplicate rows (WRT= -=20 -> the merge condition) and consequently raise a duplicate key error if=20 -> there is a UNIQUE constraint. ->=20 -> If that is what the standard describes, then it can be implemented=20 -> without any sort of index or constraint requirement. The query tree for= -=20 -> MERGE will have the INTO relation as a left outer join. In the case of a= -=20 -> match of this outer join, one set of targetlist expressions is used to=20 -> form the result tuple containing the INTO-relations ctid. That result=20 -> tuple us useable for heap_update() or heap_delete(). In the case of=20 -> no-match another set of target list expressions is used, suitable for=20 -> heap_insert(). This way, MERGE will work with one single sequential scan= -=20 -> of the INTO relation in case there is no suitable index. - -Yes, that's the way I read the standard and how I was thinking it could -be implemented. It does simplify the case suggested by people that want -atomic REPLACE, because you only have one statement to repeat until you -get success. - -do - MERGE ...; -while( not error and modified_rows <> 1 ) - -I was thinking that we could make a seperate REPLACE command which can -only insert or update one row, but can do it atomically. Basically a -loop like above with builtin savepoints. - -Alternativly, we could special case the MERGE-without-USING case, in -the cases where the plan simply devolves into an Index Scan, but I -don't like special casing. If we're going to have special semantics we -should have a seperate statement so it's clear that it's special. - -Have a nice day, ---=20 -Martijn van Oosterhout http://svana.org/kleptog/ -> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a -> tool for doing 5% of the work and then sitting around waiting for someone -> else to do the other 95% so you can sue them. - ---NklN7DEeGtkPCoo3 -Content-Type: application/pgp-signature -Content-Disposition: inline - ------BEGIN PGP SIGNATURE----- -Version: GnuPG v1.0.6 (GNU/Linux) -Comment: For info see http://www.gnupg.org - -iD8DBQFDhyXEIB7bNG8LQkwRAlxiAJ4tVMSPKRELg0H7na778cppeU1opACgkawW -xsYY7O6aMVjdXT4ye3fS6KI= -=XS8K ------END PGP SIGNATURE----- - ---NklN7DEeGtkPCoo3-- - diff --git a/doc/TODO.detail/pitr b/doc/TODO.detail/pitr deleted file mode 100644 index d4d4c41404..0000000000 --- a/doc/TODO.detail/pitr +++ /dev/null @@ -1,827 +0,0 @@ -From pgsql-admin-owner+M15281=pgman=candle.pha.pa.us@postgresql.org Thu Oct 21 18:57:36 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9LLvYf17059 - for ; Thu, 21 Oct 2004 17:57:34 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 79D9132A71A - for ; Thu, 21 Oct 2004 22:57:29 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 80515-02 for ; - Thu, 21 Oct 2004 21:57:26 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 1209432A70E - for ; Thu, 21 Oct 2004 22:57:29 +0100 (BST) -X-Original-To: pgsql-admin-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 4B39932A6C3 - for ; Thu, 21 Oct 2004 22:51:01 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 78125-02 - for ; - Thu, 21 Oct 2004 21:50:48 +0000 (GMT) -Received: from news.hub.org (news.hub.org [200.46.204.72]) - by svr1.postgresql.org (Postfix) with ESMTP id 27F0632A6C2 - for ; Thu, 21 Oct 2004 22:50:49 +0100 (BST) -Received: from news.hub.org (news.hub.org [200.46.204.72]) - by news.hub.org (8.12.9/8.12.9) with ESMTP id i9LLojJ7079086 - for ; Thu, 21 Oct 2004 21:50:45 GMT - (envelope-from news@news.hub.org) -Received: (from news@localhost) - by news.hub.org (8.12.9/8.12.9/Submit) id i9LLnd7p078783 - for pgsql-admin@postgresql.org; Thu, 21 Oct 2004 21:49:39 GMT -From: Gaetano Mendola -X-Newsgroups: comp.databases.postgresql.admin -Subject: Re: [ADMIN] replication using WAL archives -Date: Thu, 21 Oct 2004 23:49:35 +0200 -Organization: PYRENET Midi-pyrenees Provider -Lines: 216 -Message-ID: <41782EEF.5040708@bigfoot.com> -References: <002801c4b739$68450870$7201a8c0@mst1x5r347kymb> <1098384082.15573.14.camel@camel> -MIME-Version: 1.0 -Content-Type: multipart/mixed; - boundary="------------060900090803090101060101" -X-Complaints-To: abuse@pyrenet.fr -cc: iain@mst.co.jp -To: Robert Treat -User-Agent: Mozilla Thunderbird 0.8 (Windows/20040913) -X-Accept-Language: en-us, en -In-Reply-To: <1098384082.15573.14.camel@camel> -X-Enigmail-Version: 0.86.1.0 -X-Enigmail-Supports: pgp-inline, pgp-mime -To: pgsql-admin@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-admin -Precedence: bulk -Sender: pgsql-admin-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham - version=2.61 -Status: OR - -This is a multi-part message in MIME format. ---------------060900090803090101060101 -Content-Type: text/plain; charset=ISO-8859-1; format=flowed -Content-Transfer-Encoding: 7bit - -Robert Treat wrote: -> On Thu, 2004-10-21 at 02:44, Iain wrote: -> ->>Hi, ->> ->>I thought I read something about this in relation to v8, but I can't ->>find any reference to it now... is it (or will it be) possible to do ->>master-slave style database replication by transmitting log files to the ->>standby server and having it process them? ->> -> -> -> I'm not certain if this is 8.0, but some folks have created a working -> version against the 8.0 code that will do something like this. Search -> the pgsql-hacker mail list archives for more information on it. - -I sent a post on hackers, I put it here: - -======================================================================= -Hi all, -I seen that Eric Kerin did the work suggested by Tom about -how to use the PITR in order to have an hot spare postgres, -writing a C program. - -I did the same writing 2 shell scripts, one of them perform -the restore the other one deliver the partial filled wal and -check if the postmaster is alive ( check if the pid process -still exist ). - -With these two scripts I'm able to have an hot spare installation, -and the spare one go alive when the first postmaster dies. - -How test it: - -1) Master node: - modify postgresql.conf using: - -~ archive_command = 'cp %p /mnt/server/archivedir/%f' - -~ launch postgres and perform a backup as doc - -~ http://developer.postgresql.org/docs/postgres/backup-online.html - - suggest to do - - launch the script: - - partial_wal_deliver.sh /mnt/server/partialdir - -~ this script will delivery each 10 seconds the "current" wal file, -~ and touch the "alive" file in order to notify the spare node that -~ the master node is up and running - - -2) Spare node: - create a recovery.conf with the line: - -~ restore_command = 'restore.sh /mnt/server/archivedir/%f %p /mnt/server/partialdir' - -~ replace the content of data directory with the backup performed at point 1, -~ remove any file present in the pg_xlog directory ( leaving there the archive_status -~ directory ) and remove the postmaster.pid file ( this is necessary if you are running -~ the spare postgres on the same hw ). - -~ launch the postmaster, the restore will continue till the "alive" file present in the -~ /mnt/server/partialdir directory is not updated for 60 seconds ( you can modify this -~ values inside the restore.sh script ). - -Be sure that restore.sh and all directories involved are accessible - -Let me know. - - -This is a first step, of course, as Eric Kerin did, is better port these script -in C and make it more robust. - -Postgres can help this process, as suggested by Tom creating a pg_current_wal() -or even better having two new GUC parameters: archive_current_wal_command and -archive_current_wal_delay. - -I problem I discover during the tests is that if you shut down the spare node -and the restore_command is still waiting for a file then the postmaster will never -exit :-( -========================================================================== - -I hope that is clear. - - - -Regards -Gaetano Mendola - - - ---------------060900090803090101060101 -Content-Type: text/plain; - name="restore.sh" -Content-Transfer-Encoding: 7bit -Content-Disposition: inline; - filename="restore.sh" - -#!/bin/bash - - -SOURCE=$1 -TARGET=$2 -PARTIAL=$3 - -SIZE_EXPECTED=16777216 #bytes 16 MB -DIED_TIME=60 #seconds - -function test_existence -{ - if [ -f ${SOURCE} ] - then - COUNTER=0 - - #I have to check if the file is begin copied - #I assume that it will reach the right - #size in a few seconds - - while [ $(stat -c '%s' ${SOURCE} ) -lt $SIZE_EXPECTED ] - do - sleep 1 - let COUNTER+=1 - if [ 20 -lt $COUNTER ] - then - exit 1 # BAILING OUT - fi - done - - cp $SOURCE $TARGET - exit 0 - fi - echo ${SOURCE}"> not found" - - #if is looking for a history file and not exist - #I have suddenly exit - echo $SOURCE | grep history > /dev/null 2>&1 && exit 1 -} - - -while [ 1 ] -do - - test_existence - - #CHECK IF THE MASTER IS ALIVE - DELTA_TIME=$(( $( date +'%s' ) - $( stat -c '%Z' ${PARTIAL}/alive ) )) - if [ $DIED_TIME -lt $DELTA_TIME ] - then - echo "Master is dead..." - # Master is dead - CURRENT_WAL=$( basename $SOURCE ) - echo "Partial: " ${PARTIAL} - echo "Current wal: " ${CURRENT_WAL} - echo "Target: " ${TARGET} - cp ${PARTIAL}/${CURRENT_WAL}.partial ${TARGET} > /dev/null 2>&1 && exit 0 - exit 1 - fi - - sleep 1 - -done - ---------------060900090803090101060101 -Content-Type: text/plain; - name="partial_wal_deliver.sh" -Content-Transfer-Encoding: 7bit -Content-Disposition: inline; - filename="partial_wal_deliver.sh" - -#!/bin/bash - -PID=$1 -PARTIAL=$2 -PGXLOG=$3 - -function copy_last_wal -{ - FILE=$( ls -t1p $PGXLOG | grep -v / | head -1 ) - - echo "Last Wal> " $FILE - - cp ${PGXLOG}/${FILE} ${PARTIAL}/${FILE}.tmp - mv ${PARTIAL}/${FILE}.tmp ${PARTIAL}/${FILE}.partial - find ${PARTIAL} -name *.partial | grep -v ${FILE} | xargs -i rm -fr {} -} - - -while [ 1 ] -do - ps --pid $PID > /dev/null 2>&1 - ALIVE=$? - - if [ "${ALIVE}" == "1" ] - then - #The process is dead - echo "Process dead" - copy_last_wal - exit 1 - fi - - #The process still exist - touch ${PARTIAL}/alive - copy_last_wal - - sleep 10 -done - ---------------060900090803090101060101 -Content-Type: text/plain -Content-Disposition: inline -Content-Transfer-Encoding: 8bit -MIME-Version: 1.0 - - ----------------------------(end of broadcast)--------------------------- -TIP 5: Have you checked our extensive FAQ? - - http://www.postgresql.org/docs/faqs/FAQ.html - ---------------060900090803090101060101-- - -From pgsql-admin-owner+M15295=pgman=candle.pha.pa.us@postgresql.org Fri Oct 22 06:32:38 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9M9Waf18397 - for ; Fri, 22 Oct 2004 05:32:36 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 9C9A532AC61 - for ; Fri, 22 Oct 2004 10:32:32 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 53654-01 for ; - Fri, 22 Oct 2004 09:32:26 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 3132D32AC53 - for ; Fri, 22 Oct 2004 10:32:32 +0100 (BST) -X-Original-To: pgsql-admin-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id DC46E32A095 - for ; Fri, 22 Oct 2004 10:23:07 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 49812-03 - for ; - Fri, 22 Oct 2004 09:22:52 +0000 (GMT) -Received: from cmailg3.svr.pol.co.uk (cmailg3.svr.pol.co.uk [195.92.195.173]) - by svr1.postgresql.org (Postfix) with ESMTP id 5AEA6329F2F - for ; Fri, 22 Oct 2004 10:22:57 +0100 (BST) -Received: from modem-21.monkey.dialup.pol.co.uk ([217.135.208.21] helo=Nightingale) - by cmailg3.svr.pol.co.uk with smtp (Exim 4.41) - id 1CKvdM-0005eh-NO; Fri, 22 Oct 2004 10:22:53 +0100 -Message-ID: <011a01c4b818$b7370a20$06e887d9@Nightingale> -From: "Simon Riggs" -To: "Gaetano Mendola" , - "Robert Treat" , -cc: -References: <002801c4b739$68450870$7201a8c0@mst1x5r347kymb> <1098384082.15573.14.camel@camel> <41782EEF.5040708@bigfoot.com> -Subject: Re: [ADMIN] replication using WAL archives -Date: Fri, 22 Oct 2004 10:22:54 +0100 -MIME-Version: 1.0 -Content-Type: text/plain; - charset="iso-8859-1" -Content-Transfer-Encoding: 7bit -X-Priority: 3 -X-MSMail-Priority: Normal -X-Mailer: Microsoft Outlook Express 6.00.2800.1409 -X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409 -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-admin -Precedence: bulk -Sender: pgsql-admin-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham - version=2.61 -Status: OR - -> Gaetano Mendola wrote -> Postgres can help this process, as suggested by Tom creating a -pg_current_wal() -> or even better having two new GUC parameters: archive_current_wal_command -and -> archive_current_wal_delay. - -OK, we can modify the archiver to do this as well as the archive-when-full -functionality. I'd already agreed to do something similar for 8.1 - -PROPOSAL: -By default, archive_max_delay would be 10 seconds. -By default, archive_current_wal_command is not set. -If archive_current_wal_command is not set, the archiver will archive a file -using archive_command only when the file is full. -If archive_current_wal_command is set, the archiver would archive a file -whichever of these occurs first... -- it is full -- the archive_max_delay timeout occurs (default: disabled) -...as you can see I've renamed archive_current_wal_delay to reflect the fact -that there is an interaction between the current mechanism (only when full) -and this additional mechanism (no longer than X secs between log files). -With that design, if the logs are being created quickly enough, then a -partial log file is never created, only full ones. - -When an xlog file is archived because it is full, then it is sent to both -archive_current_wal_command and archive_command (in that order). When the -timeout occurs and we have a partial xlog file, it would only be sent to -archive_current_wal_command. It may also be desirable to not use -archive_command at all, only to use archive_current_wal_command. That's not -currently possible because archive_command is the switch by which all of the -archive functioanlity is enabled, so you can't actually turn this off. - -There is already a timeout feature designed into archiver for safety...so I -can make that read the GUCs, above and act accordingly. - -There is an unresolved resilience issue: if the archiver goes down (or -whatever does the partial_wal copy functionality) then it it is possible -that users will continue writing to the database and creating xlog records. -It would be up to the user to define how to handle records that had been -committed to the first database in the interim before cutover. It would also -be up to the user to shut down the first node from the second - Shoot the -Other Node in the Head, as its known. All of that is up to the second node, -and as Tom says, is "the hard part"....I'm not proposing to do anything -about that at this stage, since it is implementation dependant. - -I was thinking perhaps to move to having variable size xlog files, since -their contents are now variable - no padded records at EOF. If we did that, -then the archiver could simply issue a "switch logfile" and then the -archiver would cut in anyway to copy away the xlog. Having said that it is -lots easier just to put a blind timeout in the archiver and copy the file - -though I'm fairly uneasy about the point that we'd be ignoring the fact that -many people are still writing to it. But I propose doing the easy way.... - -Thoughts? - -= - = - = - -Gaetano - skim-reading your script, how do you handle the situation when a -new xlog file has been written within 10 seconds? That way the current file -number will have jumped by 2, so when your script looks for the "Last wal" -using head -1 it will find the N+2 and the intermediate file will never be -copied. Looks like a problem to me... - -> I problem I discover during the tests is that if you shut down the spare -node -> and the restore_command is still waiting for a file then the postmaster -will never -> exit :-( - -Hmm....Are you reporting this as a bug for 8.0? It's not on the bug list... - -Do we consider that to be desirable or not? - -Best Regards, Simon Riggs - - - ----------------------------(end of broadcast)--------------------------- -TIP 2: you can get off all lists at once with the unregister command - (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) - -From pgsql-admin-owner+M15302=pgman=candle.pha.pa.us@postgresql.org Fri Oct 22 13:56:14 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9MGuCf28637 - for ; Fri, 22 Oct 2004 12:56:13 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 54E77EAEDAA - for ; Fri, 22 Oct 2004 17:55:51 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 86116-09 for ; - Fri, 22 Oct 2004 16:55:57 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 0EC98EAEDA7 - for ; Fri, 22 Oct 2004 17:55:51 +0100 (BST) -X-Original-To: pgsql-admin-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 5DB98EAEDBE - for ; Fri, 22 Oct 2004 17:45:13 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 82473-08 - for ; - Fri, 22 Oct 2004 16:45:11 +0000 (GMT) -Received: from pns.mm.eutelsat.org (pns.mm.eutelsat.org [194.214.173.227]) - by svr1.postgresql.org (Postfix) with ESMTP id E49F0EAEDB5 - for ; Fri, 22 Oct 2004 17:45:00 +0100 (BST) -Received: from nts-03.mm.eutelsat.org (localhost [127.0.0.1]) - by pns.mm.eutelsat.org (8.11.6/linuxconf) with ESMTP id i9MGh0U26124; - Fri, 22 Oct 2004 18:43:01 +0200 -Received: from [127.0.0.1] (accesspoint.mm.eutelsat.org [194.214.173.4]) - by nts-03.mm.eutelsat.org (8.11.6/linuxconf) with ESMTP id i9MGj5f09681; - Fri, 22 Oct 2004 18:45:05 +0200 -Message-ID: <4179390B.10700@bigfoot.com> -Date: Fri, 22 Oct 2004 18:44:59 +0200 -From: Gaetano Mendola -User-Agent: Mozilla Thunderbird 0.8 (Windows/20040913) -X-Accept-Language: en-us, en -MIME-Version: 1.0 -To: Simon Riggs -cc: Robert Treat , pgsql-admin@postgresql.org, - iain@mst.co.jp -Subject: Re: [ADMIN] replication using WAL archives -References: <002801c4b739$68450870$7201a8c0@mst1x5r347kymb> <1098384082.15573.14.camel@camel> <41782EEF.5040708@bigfoot.com> <011a01c4b818$b7370a20$06e887d9@Nightingale> -In-Reply-To: <011a01c4b818$b7370a20$06e887d9@Nightingale> -X-Enigmail-Version: 0.86.1.0 -X-Enigmail-Supports: pgp-inline, pgp-mime -Content-Type: text/plain; charset=ISO-8859-1; format=flowed -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-admin -Precedence: bulk -Sender: pgsql-admin-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham - version=2.61 -Status: OR - ------BEGIN PGP SIGNED MESSAGE----- -Hash: SHA1 - -Simon Riggs wrote: -|>Gaetano Mendola wrote -|>Postgres can help this process, as suggested by Tom creating a -| -| pg_current_wal() -| -|>or even better having two new GUC parameters: archive_current_wal_command -| -| and -| -|>archive_current_wal_delay. -| -| -| OK, we can modify the archiver to do this as well as the archive-when-full -| functionality. I'd already agreed to do something similar for 8.1 -| -| PROPOSAL: -| By default, archive_max_delay would be 10 seconds. -| By default, archive_current_wal_command is not set. -| If archive_current_wal_command is not set, the archiver will archive a file -| using archive_command only when the file is full. -| If archive_current_wal_command is set, the archiver would archive a file -| whichever of these occurs first... -| - it is full -| - the archive_max_delay timeout occurs (default: disabled) -| ...as you can see I've renamed archive_current_wal_delay to reflect the fact -| that there is an interaction between the current mechanism (only when full) -| and this additional mechanism (no longer than X secs between log files). -| With that design, if the logs are being created quickly enough, then a -| partial log file is never created, only full ones. -| -| When an xlog file is archived because it is full, then it is sent to both -| archive_current_wal_command and archive_command (in that order). When the -| timeout occurs and we have a partial xlog file, it would only be sent to -| archive_current_wal_command. It may also be desirable to not use -| archive_command at all, only to use archive_current_wal_command. That's not -| currently possible because archive_command is the switch by which all of the -| archive functioanlity is enabled, so you can't actually turn this off. - -It seems good to me, the script behind archive command can be a nop if someone -want use the archive_current_wal_command - - -| = - = - = -| -| Gaetano - skim-reading your script, how do you handle the situation when a -| new xlog file has been written within 10 seconds? That way the current file -| number will have jumped by 2, so when your script looks for the "Last wal" -| using head -1 it will find the N+2 and the intermediate file will never be -| copied. Looks like a problem to me... - - -Yes, the only window failure I seen ( but I don't know if it's possible ) - -Master: -~ log N created - log N filled - archive log N - log N+1 created - log N+1 filled -~ log N+2 created -~ <---- the master die here before to archive the log N+1 -~ archive log N+1 - - -in this case as you underline tha last log archived is the N and the N+2 -partial wal is added to archived wal collection and in the recovery fase -the recovery stop after processing the log N. - -Is it possible that the postmaster create the N+2 file without finish to archive -the N+1 ? ( I suspect yes :-( ) - -The only cure I see here is to look for not archived WAL ( if possible ). - - -|>I problem I discover during the tests is that if you shut down the spare -|>node and the restore_command is still waiting for a file then the postmaster -|>will never exit :-( -| -| -| Hmm....Are you reporting this as a bug for 8.0? It's not on the bug list... - -For me is a behave to avoid. - - - -Regards -Gaetano Mendola - - - - - - - ------BEGIN PGP SIGNATURE----- -Version: GnuPG v1.2.5 (MingW32) -Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org - -iD8DBQFBeTkJ7UpzwH2SGd4RAsMxAKCbV7W+wrGBocf2Ftlthm0egAlIWACgp87L -KU/YusyHuvT7jSFwZVKpP3M= -=rWZx ------END PGP SIGNATURE----- - - ----------------------------(end of broadcast)--------------------------- -TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org - -From pgsql-admin-owner+M15303=pgman=candle.pha.pa.us@postgresql.org Fri Oct 22 14:43:36 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9MHhZf06453 - for ; Fri, 22 Oct 2004 13:43:35 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 01DD2EADBB7 - for ; Fri, 22 Oct 2004 18:43:13 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 01872-03 for ; - Fri, 22 Oct 2004 17:43:19 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 9E633EADAD4 - for ; Fri, 22 Oct 2004 18:43:12 +0100 (BST) -X-Original-To: pgsql-admin-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id C1133EAED89 - for ; Fri, 22 Oct 2004 18:31:20 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 97130-03 - for ; - Fri, 22 Oct 2004 17:31:17 +0000 (GMT) -Received: from cmailm2.svr.pol.co.uk (cmailm2.svr.pol.co.uk [195.92.193.210]) - by svr1.postgresql.org (Postfix) with ESMTP id 276CAEADBBD - for ; Fri, 22 Oct 2004 18:31:07 +0100 (BST) -Received: from modem-558.snake.dialup.pol.co.uk ([62.137.114.46] helo=[192.168.0.102]) - by cmailm2.svr.pol.co.uk with esmtp (Exim 4.41) - id 1CL3G3-0001Tx-K5; Fri, 22 Oct 2004 18:31:20 +0100 -Subject: Re: [ADMIN] replication using WAL archives -From: Simon Riggs -To: Gaetano Mendola -cc: Robert Treat , pgsql-admin@postgresql.org, - iain@mst.co.jp -In-Reply-To: <4179390B.10700@bigfoot.com> -References: <002801c4b739$68450870$7201a8c0@mst1x5r347kymb> - <1098384082.15573.14.camel@camel> <41782EEF.5040708@bigfoot.com> - <011a01c4b818$b7370a20$06e887d9@Nightingale> <4179390B.10700@bigfoot.com> -Content-Type: text/plain -Organization: 2nd Quadrant -Message-ID: <1098466150.20926.13.camel@localhost.localdomain> -MIME-Version: 1.0 -X-Mailer: Ximian Evolution 1.4.6 (1.4.6-2) -Date: Fri, 22 Oct 2004 18:29:10 +0100 -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-admin -Precedence: bulk -Sender: pgsql-admin-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham - version=2.61 -Status: OR - -On Fri, 2004-10-22 at 17:44, Gaetano Mendola wrote: -> | Gaetano - skim-reading your script, how do you handle the situation when a -> | new xlog file has been written within 10 seconds? That way the current file -> | number will have jumped by 2, so when your script looks for the "Last wal" -> | using head -1 it will find the N+2 and the intermediate file will never be -> | copied. Looks like a problem to me... -> -> -> Yes, the only window failure I seen ( but I don't know if it's possible ) -> -> Master: -> ~ log N created -> log N filled -> archive log N -> log N+1 created -> log N+1 filled -> ~ log N+2 created -> ~ <---- the master die here before to archive the log N+1 -> ~ archive log N+1 -> -> -> in this case as you underline tha last log archived is the N and the N+2 -> partial wal is added to archived wal collection and in the recovery fase -> the recovery stop after processing the log N. -> -> Is it possible that the postmaster create the N+2 file without finish to archive -> the N+1 ? ( I suspect yes :-( ) -> -> The only cure I see here is to look for not archived WAL ( if possible ). -> - -Hmm...well you aren't looking for archived wal, you're just looking at -wal...which is a different thing... - -Situation I thought I saw was: - -- copy away current partial filled xlog N -- xlog N fills, N+1 starts -- xlog N+1 fills, N+2 starts -- copy away current partial filled xlog: N+2 (+10 secs later) - -i.e. if time to fill xlog (is ever) < time to copy away current xlog, -then you miss one. - -So problem: you can miss one and never know you've missed one until the -recovery can't find it, which it never returns from...so it just hangs. - -[Just so we're all clear: we're talking about Gaetano's script, not the -PostgreSQL archver. The postgresql archiver doesn't do it that way, so -it never misses one.] - --- -Best Regards, Simon Riggs - - ----------------------------(end of broadcast)--------------------------- -TIP 9: the planner will ignore your desire to choose an index scan if your - joining column's datatypes do not match - -From pgsql-admin-owner+M15306=pgman=candle.pha.pa.us@postgresql.org Fri Oct 22 17:56:07 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9MKu6f05264 - for ; Fri, 22 Oct 2004 16:56:06 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 4F4C2EAE4AE - for ; Fri, 22 Oct 2004 21:55:41 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 62857-05 for ; - Fri, 22 Oct 2004 20:55:48 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 095CEEAE4AC - for ; Fri, 22 Oct 2004 21:55:41 +0100 (BST) -X-Original-To: pgsql-admin-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 3FC9BEAE486 - for ; Fri, 22 Oct 2004 21:50:48 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 62565-02 - for ; - Fri, 22 Oct 2004 20:50:48 +0000 (GMT) -Received: from news.hub.org (news.hub.org [200.46.204.72]) - by svr1.postgresql.org (Postfix) with ESMTP id 06C49EAE46B - for ; Fri, 22 Oct 2004 21:50:40 +0100 (BST) -Received: from news.hub.org (news.hub.org [200.46.204.72]) - by news.hub.org (8.12.9/8.12.9) with ESMTP id i9MKolJB062812 - for ; Fri, 22 Oct 2004 20:50:48 GMT - (envelope-from news@news.hub.org) -Received: (from news@localhost) - by news.hub.org (8.12.9/8.12.9/Submit) id i9MKoRHh062731 - for pgsql-admin@postgresql.org; Fri, 22 Oct 2004 20:50:27 GMT -From: Gaetano Mendola -X-Newsgroups: comp.databases.postgresql.admin -Subject: Re: [ADMIN] replication using WAL archives -Date: Fri, 22 Oct 2004 22:50:34 +0200 -Organization: PYRENET Midi-pyrenees Provider -Lines: 39 -Message-ID: <4179729A.5020401@bigfoot.com> -References: <002801c4b739$68450870$7201a8c0@mst1x5r347kymb> <1098384082.15573.14.camel@camel> <41782EEF.5040708@bigfoot.com> <011a01c4b818$b7370a20$06e887d9@Nightingale> <4179390B.10700@bigfoot.com> <1098466150.20926.13.camel@localhost.localdomain> -MIME-Version: 1.0 -Content-Type: text/plain; charset=ISO-8859-1; format=flowed -Content-Transfer-Encoding: 7bit -X-Complaints-To: abuse@pyrenet.fr -To: Simon Riggs -User-Agent: Mozilla Thunderbird 0.8 (Windows/20040913) -X-Accept-Language: en-us, en -In-Reply-To: <1098466150.20926.13.camel@localhost.localdomain> -X-Enigmail-Version: 0.86.1.0 -X-Enigmail-Supports: pgp-inline, pgp-mime -To: pgsql-admin@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-admin -Precedence: bulk -Sender: pgsql-admin-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham - version=2.61 -Status: OR - -Simon Riggs wrote: - - > Situation I thought I saw was: - > - > - copy away current partial filled xlog N - > - xlog N fills, N+1 starts - > - xlog N+1 fills, N+2 starts - > - copy away current partial filled xlog: N+2 (+10 secs later) - > - > i.e. if time to fill xlog (is ever) < time to copy away current xlog, - > then you miss one. - > - > So problem: you can miss one and never know you've missed one until the - > recovery can't find it, which it never returns from...so it just hangs. - -No. The restore.sh is not smart enough to know the last wal that must be -replayed, the only "smart thing" is to copy the supposed "current wal" in the -archive directory. - -The script hang (and is a feature not a bug) if and only if the master is alive -( at least I'm not seeing any other hang ). - -In your example in the archived directory will be present the files until logN -and logN+2 ( the current wal ) is in the partial directory, if the master die, -the restore.sh will copy logN+2 in the archived directory, the spare node will -execute restore.sh with file logN+1 as argument and if is not found then the -restore.sh will exit. - - -Regards -Gaetano Mendola - - - - - - - - - ----------------------------(end of broadcast)--------------------------- -TIP 8: explain analyze is your friend - diff --git a/doc/TODO.detail/psql b/doc/TODO.detail/psql deleted file mode 100644 index 5f621d4cc3..0000000000 --- a/doc/TODO.detail/psql +++ /dev/null @@ -1,1725 +0,0 @@ -From pgsql-hackers-owner+M48909@postgresql.org Thu Jan 8 21:54:03 2004 -Return-path: -Received: from noon.pghoster.com (noon.pghoster.com [64.246.0.64]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i092s2X02439 - for ; Thu, 8 Jan 2004 21:54:02 -0500 (EST) -Received: from svr1.postgresql.org ([200.46.204.71] helo=postgresql.org) - by noon.pghoster.com with esmtp (Exim 4.24) - id 1AemmG-0002Wx-5B; Thu, 08 Jan 2004 20:53:36 -0600 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (neptune.hub.org [200.46.204.2]) - by svr1.postgresql.org (Postfix) with ESMTP id 88703D1B46E - for ; Mon, 5 Jan 2004 02:00:10 +0000 (GMT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) - with ESMTP id 72572-01 - for ; - Sun, 4 Jan 2004 21:59:23 -0400 (AST) -Received: from rwcrmhc13.comcast.net (rwcrmhc13.comcast.net [204.127.198.39]) - by svr1.postgresql.org (Postfix) with ESMTP id DD336D1B454 - for ; Sun, 4 Jan 2004 21:59:04 -0400 (AST) -Received: from lorenso.com (c-24-1-26-144.client.comcast.net[24.1.26.144]) - by comcast.net (rwcrmhc13) with ESMTP - id <20040105015908015005cvvee>; Mon, 5 Jan 2004 01:59:08 +0000 -Message-ID: <3FF8C4E6.9090008@lorenso.com> -Date: Sun, 04 Jan 2004 19:59:02 -0600 -From: "D. Dante Lorenso" -User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6b) Gecko/20031205 Thunderbird/0.4 -X-Accept-Language: en-us, en -MIME-Version: 1.0 -To: "Alex J. Avriette" -cc: Bruce Momjian , - PostgreSQL-development -Subject: Re: [HACKERS] psql \d option list overloaded -References: <3FEE6DFB.9040408@lorenso.com> <200401040125.i041PLR14687@candle.pha.pa.us> <20040104191322.GD8524@posixnap.net> -In-Reply-To: <20040104191322.GD8524@posixnap.net> -Content-Type: text/plain; charset=ISO-8859-1; format=flowed -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by amavisd-new at postgresql.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-AntiAbuse: This header was added to track abuse, please include it with any abuse report -X-AntiAbuse: Primary Hostname - noon.pghoster.com -X-AntiAbuse: Original Domain - candle.pha.pa.us -X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] -X-AntiAbuse: Sender Address Domain - postgresql.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.8 required=5.0 tests=BAYES_00,HTML_MESSAGE - autolearn=no version=2.61 -Status: OR - -Alex J. Avriette wrote: - ->On Sat, Jan 03, 2004 at 08:25:21PM -0500, Bruce Momjian wrote: -> -> ->>>I finally figure it out, I just end up forgetting again later. I still ->>>have no clue how I'd find the same data without using psql. In MySQL ->>>I can run those queries from PHP, PERL...etc. I know you can find that ->>>data in system tables in PostgreSQL, but I don't wanna muck around with ->>>all that. I just wanna do something as simple as MySQL. ->>> ->>> ->>[ Moved to hackers.] ->> ->>I am starting to agree that our \d* handling is just too overloaded. ->>Look at the option list from \?: ->> ->> ->>I like the idea of adding a new syntax to show that information using ->>simple SQL command syntax, and putting it in the backend so all ->>applications can access it. I know we have information schema, and ->>maybe that can be used to make this simpler. ->> ->> ->Bruce, while I agree with you about \d (and all its children), as well ->as the querying we talked about on irc, I disagree with the notion of a ->"SHOW DATABASES" query. This is one of the things that irritates me ->about mysql is the pseudo-sql that everyone has come to accept ... It doesn't ->make sense to create pseudo-sql, when all you're abstracting is function-macros... -> -Anything other than simple, short commands is a waste, IMHO. I can easily -remember SHOW DATABASES and SHOW TABLES and DESC , because they -reflect -my intensions directly and 'make sense'. - -Using the slash commands works if you are familiar with them ... sorta -like 'ls' switches (I type 'ls -alF' without thinking about what those -switches do because it's embedded in my head from years of repetition. -Any other flags to 'ls', and I gotta go hit the man pages.) - -What's more important is the ability to use these commands from any -interface not just 'psql' client. I think 'psql' already has the slash -commands. No need to create NEW slash commands there... - ->If you want to find out how to show the databases in sql, use psql -E. -> -> -Have you actually done that? OMG! - -1) Using System Catalogs ... (from psql -E) - - SELECT n.nspname as "Schema", - c.relname as "Name", - CASE c.relkind - WHEN 'r' THEN 'table' - WHEN 'v' THEN 'view' - WHEN 'i' THEN 'index' - WHEN 'S' THEN 'sequence' - WHEN 's' THEN 'special' END as "Type", - u.usename as "Owner" - FROM pg_catalog.pg_class c - LEFT JOIN pg_catalog.pg_user u ON u.usesysid = c.relowner - LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace - WHERE c.relkind IN ('r','') - AND n.nspname NOT IN ('pg_catalog', 'pg_toast') - AND pg_catalog.pg_table_is_visible(c.oid) - ORDER BY 1,2; - -or ... - -2) (using information schema ... little better) - - SELECT table_name FROM information_schema.tables WHERE table_schema -= 'public'; - -or ... - -3) like MySQL does it... - - SHOW TABLES; - -Lemme think about which one I prefer ;-) Uh, Ok, I'm done thinking -now. hehe. - -There's something to be said about the 'SHOW'and 'DESC' sql-extensions -added into MySQL. Newbies can really 'get' it quickly. It's what really -sold me on MySQL when I first learned it. For me, it's like: - - 'dir' in DOS, - 'ls' in Unix - 'SHOW' in MySQL - ??? in PostgreSQL ? - -Sure, with time as my database needs grew and I matured as a developer, -I eventually gained more respect for PostgreSQL and have made the switch -even without this feature, but to this day, I really think MySQL *did it -right* with those extensions. You can't become a PostgreSQL guru without -being a newbie first. I vote we make it easier for newbies. - -Dante - - - ----------------------------(end of broadcast)--------------------------- -TIP 5: Have you checked our extensive FAQ? - - http://www.postgresql.org/docs/faqs/FAQ.html - -From pgsql-hackers-owner+M48908@postgresql.org Thu Jan 8 21:50:03 2004 -Return-path: -Received: from noon.pghoster.com (noon.pghoster.com [64.246.0.64]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i092ntX01459 - for ; Thu, 8 Jan 2004 21:50:03 -0500 (EST) -Received: from svr1.postgresql.org ([200.46.204.71] helo=postgresql.org) - by noon.pghoster.com with esmtp (Exim 4.24) - id 1AemiO-0002D1-Di; Thu, 08 Jan 2004 20:49:36 -0600 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (neptune.hub.org [200.46.204.2]) - by svr1.postgresql.org (Postfix) with ESMTP id 7FD9BD1B473 - for ; Mon, 5 Jan 2004 02:05:14 +0000 (GMT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) - with ESMTP id 70484-09 - for ; - Sun, 4 Jan 2004 22:04:28 -0400 (AST) -Received: from news.hub.org (news.hub.org [200.46.204.72]) - by svr1.postgresql.org (Postfix) with ESMTP id 8F8CAD1BB73 - for ; Sun, 4 Jan 2004 22:04:16 -0400 (AST) -Received: from news.hub.org (news.hub.org [200.46.204.72]) - by news.hub.org (8.12.9/8.12.9) with ESMTP id i0524DU6041774 - for ; Mon, 5 Jan 2004 02:04:13 GMT - (envelope-from news@news.hub.org) -Received: (from news@localhost) - by news.hub.org (8.12.9/8.12.9/Submit) id i0521bd7040362 - for pgsql-hackers@postgresql.org; Mon, 5 Jan 2004 02:01:37 GMT -From: "William ZHANG" -X-Newsgroups: comp.databases.postgresql.hackers -Subject: Re: [HACKERS] psql \d option list overloaded -Date: Mon, 5 Jan 2004 10:00:55 +0800 -Organization: N/A -Lines: 6 -Message-ID: -References: <3FEE6DFB.9040408@lorenso.com> <200401040125.i041PLR14687@candle.pha.pa.us> -X-Complaints-To: usenet@news.hub.org -X-Priority: 3 -X-MSMail-Priority: Normal -X-Newsreader: Microsoft Outlook Express 6.00.2800.1106 -X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 -To: pgsql-hackers@postgresql.org -X-Virus-Scanned: by amavisd-new at postgresql.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-AntiAbuse: This header was added to track abuse, please include it with any abuse report -X-AntiAbuse: Primary Hostname - noon.pghoster.com -X-AntiAbuse: Original Domain - candle.pha.pa.us -X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] -X-AntiAbuse: Sender Address Domain - postgresql.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-3.7 required=5.0 tests=BAYES_00,PRIORITY_NO_NAME - autolearn=no version=2.61 -Status: OR - -I think moving the \d and simliar features in psql -to SQL is a good idea. That will make the features -available in any client library. As for the syntax, -maybe a investigation is needed. - - - ----------------------------(end of broadcast)--------------------------- -TIP 6: Have you searched our list archives? - - http://archives.postgresql.org - -From pgsql-hackers-owner+M48835=pgman=candle.pha.pa.us@postgresql.org Tue Jan 6 03:08:59 2004 -Return-path: -Received: from noon.pghoster.com (noon.pghoster.com [64.246.0.64]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i0688wX03365 - for ; Tue, 6 Jan 2004 03:08:59 -0500 (EST) -Received: from svr1.postgresql.org ([200.46.204.71] helo=postgresql.org) - by noon.pghoster.com with esmtp (Exim 4.24) - id 1AdmGn-0006qB-5D - for pgman@candle.pha.pa.us; Tue, 06 Jan 2004 02:08:57 -0600 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (neptune.hub.org [200.46.204.2]) - by svr1.postgresql.org (Postfix) with ESMTP id 6D470D1B446 - for ; Tue, 6 Jan 2004 08:02:13 +0000 (GMT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) - with ESMTP id 19966-06 - for ; - Tue, 6 Jan 2004 04:01:30 -0400 (AST) -Received: from mail.eckpart.de (unknown [62.206.85.106]) - by svr1.postgresql.org (Postfix) with SMTP id C278CD1B46E - for ; Tue, 6 Jan 2004 04:01:06 -0400 (AST) -Received: (qmail 349 invoked from network); 6 Jan 2004 08:01:11 -0000 -Received: from unknown (HELO at13.eckpart.de) (192.168.41.70) - by cserv.eckpart.de with SMTP; 6 Jan 2004 08:01:11 -0000 -From: Tommi Maekitalo -Organization: Dr. Eckhardt + Partner GmbH -To: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] psql \d option list overloaded -Date: Tue, 6 Jan 2004 09:01:10 +0100 -User-Agent: KMail/1.5.4 -References: <3FEE6DFB.9040408@lorenso.com> <200401040125.i041PLR14687@candle.pha.pa.us> <20040104191322.GD8524@posixnap.net> -In-Reply-To: <20040104191322.GD8524@posixnap.net> -MIME-Version: 1.0 -Content-Type: text/plain; - charset="iso-8859-1" -Content-Transfer-Encoding: 7bit -Content-Disposition: inline -Message-ID: <200401060901.10830.t.maekitalo@epgmbh.de> -X-Virus-Scanned: by amavisd-new at postgresql.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-AntiAbuse: This header was added to track abuse, please include it with any abuse report -X-AntiAbuse: Primary Hostname - noon.pghoster.com -X-AntiAbuse: Original Domain - candle.pha.pa.us -X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] -X-AntiAbuse: Sender Address Domain - postgresql.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham - version=2.61 -Status: OR - -Am Sonntag, 4. Januar 2004 20:13 schrieb Alex J. Avriette: -> On Sat, Jan 03, 2004 at 08:25:21PM -0500, Bruce Momjian wrote: -> > > I finally figure it out, I just end up forgetting again later. I still -... -> -> /functions -> /databases -> -... - -Long options sounds really good. It is like GNU-tools. A single - for single -character options and a double -- for long options. - -Ah - a single \ for short options in postgresql and a double \\ for long? What -do you think? - - --- -Dr. Eckhardt + Partner GmbH -http://www.epgmbh.de - - ----------------------------(end of broadcast)--------------------------- -TIP 7: don't forget to increase your free space map settings - -From pgsql-hackers-owner+M48912@postgresql.org Thu Jan 8 22:37:54 2004 -Return-path: -Received: from hosting.commandprompt.com (216.commandprompt.com [207.173.200.216]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i093bpX02244 - for ; Thu, 8 Jan 2004 22:37:52 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by hosting.commandprompt.com (8.11.6/8.11.6) with ESMTP id i093U0k32213; - Thu, 8 Jan 2004 19:30:33 -0800 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (neptune.hub.org [200.46.204.2]) - by svr1.postgresql.org (Postfix) with ESMTP id 95E70D1B43E - for ; Fri, 9 Jan 2004 03:29:43 +0000 (GMT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) - with ESMTP id 28908-10 - for ; - Thu, 8 Jan 2004 23:28:57 -0400 (AST) -Received: from mail.hive.nj2.inquent.com (mc.carriermail.com [205.178.180.9]) - by svr1.postgresql.org (Postfix) with SMTP id E42E8D1B48A - for ; Thu, 8 Jan 2004 23:27:26 -0400 (AST) -Received: (qmail 28537 invoked from network); 9 Jan 2004 03:27:33 -0000 -Received: from unknown (HELO ?192.168.1.199?) (134.22.68.14) - by 205.178.180.9 with SMTP; 9 Jan 2004 03:27:33 -0000 -Subject: Re: [HACKERS] psql \d option list overloaded -From: Rod Taylor -To: "D. Dante Lorenso" -cc: "Alex J. Avriette" , - Bruce Momjian , - PostgreSQL Development -In-Reply-To: <3FF8C4E6.9090008@lorenso.com> -References: <3FEE6DFB.9040408@lorenso.com> - <200401040125.i041PLR14687@candle.pha.pa.us> - <20040104191322.GD8524@posixnap.net> <3FF8C4E6.9090008@lorenso.com> -Content-Type: text/plain -Message-ID: <1073618847.322.29.camel@jester> -MIME-Version: 1.0 -X-Mailer: Ximian Evolution 1.4.5 -Date: Thu, 08 Jan 2004 22:27:28 -0500 -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by amavisd-new at postgresql.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.8 required=5.0 tests=BAYES_00,HTML_MESSAGE - autolearn=no version=2.61 -Status: OR - -> Anything other than simple, short commands is a waste, IMHO. I can easily -> remember SHOW DATABASES and SHOW TABLES and DESC
, because they -> reflect -> my intensions directly and 'make sense'. - -Can you remember how to get a list of indexes on a particular table? How -about a specific indexes build? I ask, because I constantly forgot both -of those (don't like FROM). - -> 2) (using information schema ... little better) -> -> SELECT table_name FROM information_schema.tables WHERE table_schema -> = 'public'; -> -> or ... -> -> 3) like MySQL does it... -> -> SHOW TABLES; -> -> Lemme think about which one I prefer ;-) Uh, Ok, I'm done thinking -> now. hehe. - -I actually prefer #2 myself. It works on a number of databases aside -from just PostgreSQL. So, as a user who worked in a mixed environment it -was easier to remember. - -But I get your point. - -> Sure, with time as my database needs grew and I matured as a developer, -> I eventually gained more respect for PostgreSQL and have made the switch -> even without this feature, but to this day, I really think MySQL *did it -> right* with those extensions. You can't become a PostgreSQL guru without - -I agree with the simple SHOW TABLES command but disagree with: - -SHOW [FULL] COLUMNS FROM tbl_name [FROM db_name] [LIKE wild] - -I much prefer: - -SELECT * FROM COLUMNS WHERE table LIKE '%tab%' AND database = 'billing'; - -It's not much longer, certainly more natural to those that know SQL, and -infinitely more useful since you can create result sets that the -programmer of SHOW hadn't considered. A perfect example is the addition -of the FULL clause in SHOW. The above select does not need additional -keywords for different formatting options as it can simply use "natural" -SQL styling. - - ----------------------------(end of broadcast)--------------------------- -TIP 2: you can get off all lists at once with the unregister command - (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) - -From tswan@idigx.com Fri Jan 9 02:07:40 2004 -Return-path: -Received: from stubee.d2hosting.net (d2hosting.net [66.70.41.160]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i0977cX28507 - for ; Fri, 9 Jan 2004 02:07:40 -0500 (EST) -Received: from idigx.com (adsl-159-238-227.mob.bellsouth.net [68.159.238.227]) - by stubee.d2hosting.net (8.11.6/linuxconf) with ESMTP id i0977Qn08421; - Fri, 9 Jan 2004 01:07:26 -0600 -Message-ID: <3FFE532C.2090503@idigx.com> -Date: Fri, 09 Jan 2004 01:07:24 -0600 -From: Thomas Swan -User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6b) Gecko/20031205 Thunderbird/0.4 -X-Accept-Language: en-us, en -MIME-Version: 1.0 -To: pgsql-hackers@postgresql.org -cc: Bruce Momjian -Subject: Re: [HACKERS] psql \d option list overloaded -References: <200401060504.i0654B012562@candle.pha.pa.us> -In-Reply-To: <200401060504.i0654B012562@candle.pha.pa.us> -X-Enigmail-Version: 0.82.5.0 -X-Enigmail-Supports: pgp-inline, pgp-mime -Content-Type: text/plain; charset=ISO-8859-1 -Content-Transfer-Encoding: 7bit -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.8 required=5.0 tests=BAYES_00,HTML_MESSAGE - autolearn=ham version=2.61 -Status: OR - -Bruce Momjian wrote: - ->Alex J. Avriette wrote: -> -> ->>On Sun, Jan 04, 2004 at 07:59:02PM -0600, D. Dante Lorenso wrote: ->> ->> ->> ->>>Anything other than simple, short commands is a waste, IMHO. I can easily ->>>remember SHOW DATABASES and SHOW TABLES and DESC
, because they ->>>reflect ->>>my intensions directly and 'make sense'. ->>> ->>> ->>What makes sense to me in csh doesn't make sense in a bourne shell. ->>You can't expect all applications to work correctly. I'd like to second ->>Peter's "yep" when asked if he could remember all the various \d* ->>commands. It really comes down to whether you're trying. New software ->>(even though you may have been using it for a year) requires some ->>adjustment. ->> ->> -> ->OK, I will drop the idea. Thanks. -> -> -> -Bruce, - -The idea is not without merit. What you are looking at is a way to get -this information as a query without having to know all the intricasies -of all the pg_* internals or duplicating complex queries. "psql -E" -shows you just how tricky this is. Secondly, if this information -changes in a release, then the end user has to rewrite all of the -queries to work. Being able to issue a query to the dbms and get the -information as a normal SQL result makes sense and is definately convenient. - -The \d* commands work from psql but not from anywhere else. Try -getting the information from a PHP script by sending a "\dS" query. It -doesn't work. If the same queries were stored in the backend and -referenced by psql and also could be referenced by other scripts, this -would be a good thing and keep the work centralized. If the queries -were in the backend, the psql users could keep the \dS command but it -would call an internal function or execute a queried stored in the -system tables. - - -One option is to get the information via a function like - -SELECT * FROM pg_info('tables'); -SELECT * FROM pg_info('indexes'); - - -"psql -E" would show the same query being executed for "\dt" - -Another option if no one wanted a language construct, perhaps one option -would be to store the queries themselves in a table like pg_queries. -This also has the advantage of exposing the queries used so that they -can used as examples for other purposes. - -+------------+------------------------------------------+ -|pg_info_type|pg_query | -+------------+------------------------------------------+ -|tables |SELECT n.nspname as "Schema", c.relname | -| |as "Name", CASE c.relkind WHEN 'r' THEN | -| |'table' WHEN 'v' THEN 'view' WHEN 'i' THEN| -| |'index' WHEN 'S' THEN 'sequence' WHEN 's' | -| |THEN 'special' END as "Type", u.usename as| -| |"Owner" FROM pg_catalog.pg_class c LEFT | -| |JOIN pg_catalog.pg_user u ON u.usesysid = | -| |c.relowner LEFT JOIN | -| |pg_catalog.pg_namespace n ON n.oid = | -| |c.relnamespace WHERE c.relkind IN ('r','')| -| |AND n.nspname NOT IN ('pg_catalog', | -| |'pg_toast') AND | -| |pg_catalog.pg_table_is_visible(c.oid) | -| |ORDER BY 1,2; | -+------------+------------------------------------------+ -|indexes |SELECT n.nspname as "Schema", c.relname as| -| |"Name", CASE c.relkind WHEN 'r' THEN | -| |'table' WHEN 'v' THEN 'view' WHEN 'i' THEN| -| |'index' WHEN 'S' THEN 'sequence' WHEN 's' | -| |THEN 'special' END as "Type", u.usename as| -| |"Owner", c2.relname as "Table" FROM | -| |pg_catalog.pg_class c JOIN | -| |pg_catalog.pg_index i ON i.indexrelid = | -| |c.oid JOIN pg_catalog.pg_class c2 ON | -| |i.indrelid = c2.oid LEFT JOIN | -| |pg_catalog.pg_user u ON u.usesysid = | -| |c.relowner LEFT JOIN | -| |pg_catalog.pg_namespace n ON n.oid = | -| |c.relnamespace WHERE c.relkind IN ('i','')| -| |AND n.nspname NOT IN ('pg_catalog', | -| |'pg_toast') AND | -| |pg_catalog.pg_table_is_visible(c.oid) | -| |ORDER BY 1,2; | -+------------+------------------------------------------+ - - -Again, this is just food for thought. Perhaps it is a way to satisfy -both arguments. - -Thomas - -From pgsql-hackers-owner+M48922@postgresql.org Fri Jan 9 05:23:03 2004 -Return-path: -Received: from noon.pghoster.com (noon.pghoster.com [64.246.0.64]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i09AN1X10448 - for ; Fri, 9 Jan 2004 05:23:02 -0500 (EST) -Received: from svr1.postgresql.org ([200.46.204.71] helo=postgresql.org) - by noon.pghoster.com with esmtp (Exim 4.24) - id 1AetmQ-000637-HX; Fri, 09 Jan 2004 04:22:14 -0600 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (neptune.hub.org [200.46.204.2]) - by svr1.postgresql.org (Postfix) with ESMTP id DCA3ED1B447 - for ; Fri, 9 Jan 2004 10:20:50 +0000 (GMT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) - with ESMTP id 95279-02 - for ; - Fri, 9 Jan 2004 06:20:04 -0400 (AST) -Received: from mail.eckpart.de (unknown [62.206.85.106]) - by svr1.postgresql.org (Postfix) with SMTP id 34AE7D1B4C6 - for ; Fri, 9 Jan 2004 06:19:45 -0400 (AST) -Received: (qmail 21196 invoked from network); 9 Jan 2004 10:19:46 -0000 -Received: from unknown (HELO at13.eckpart.de) (192.168.41.70) - by cserv.eckpart.de with SMTP; 9 Jan 2004 10:19:46 -0000 -From: Tommi Maekitalo -Organization: Dr. Eckhardt + Partner GmbH -To: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] psql \d option list overloaded -Date: Fri, 9 Jan 2004 11:19:45 +0100 -User-Agent: KMail/1.5.4 -References: <3FEE6DFB.9040408@lorenso.com> <20040104191322.GD8524@posixnap.net> <3FF8C4E6.9090008@lorenso.com> -In-Reply-To: <3FF8C4E6.9090008@lorenso.com> -MIME-Version: 1.0 -Content-Type: text/plain; - charset="iso-8859-1" -Content-Transfer-Encoding: 7bit -Content-Disposition: inline -Message-ID: <200401091119.45778.t.maekitalo@epgmbh.de> -X-Virus-Scanned: by amavisd-new at postgresql.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-AntiAbuse: This header was added to track abuse, please include it with any abuse report -X-AntiAbuse: Primary Hostname - noon.pghoster.com -X-AntiAbuse: Original Domain - candle.pha.pa.us -X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] -X-AntiAbuse: Sender Address Domain - postgresql.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham - version=2.61 -Status: OR - -Hi, - -> -> 2) (using information schema ... little better) -> -> SELECT table_name FROM information_schema.tables WHERE table_schema -> = 'public'; -> -> or ... -> -... - -I just looked at the information_schema. It is a very nice feature, but -difficult to use in psql. - -I just wanted to see, what I can find here. After trying and rtfm I ended in -'\d information_schema.*'. I get a very large page wich is quite unreadable. -'\d' is normally very usable. - -It would be better not to show the view-definition. - -What if \d on views just show the column, type and attribute. \d+ would show -the full view-definition. - - -Tommi - --- -Dr. Eckhardt + Partner GmbH -http://www.epgmbh.de - - ----------------------------(end of broadcast)--------------------------- -TIP 5: Have you checked our extensive FAQ? - - http://www.postgresql.org/docs/faqs/FAQ.html - -From pgsql-hackers-owner+M48946@postgresql.org Sat Jan 10 07:42:08 2004 -Return-path: -Received: from noon.pghoster.com (noon.pghoster.com [64.246.0.64]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i0ACg6X18515 - for ; Sat, 10 Jan 2004 07:42:07 -0500 (EST) -Received: from svr1.postgresql.org ([200.46.204.71] helo=postgresql.org) - by noon.pghoster.com with esmtp (Exim 4.24) - id 1AfINd-0006bw-4f; Sat, 10 Jan 2004 06:38:17 -0600 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (neptune.hub.org [200.46.204.2]) - by svr1.postgresql.org (Postfix) with ESMTP id 35C29D1D542 - for ; Sat, 10 Jan 2004 12:36:36 +0000 (GMT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) - with ESMTP id 91943-10 - for ; - Sat, 10 Jan 2004 08:36:07 -0400 (AST) -Received: from filer (c-24-6-183-218.client.comcast.net [24.6.183.218]) - by svr1.postgresql.org (Postfix) with ESMTP id 4ACFBD1D54E - for ; Sat, 10 Jan 2004 08:36:03 -0400 (AST) -Received: from localhost (localhost [127.0.0.1]) - (uid 1000) - by filer with local; Sat, 10 Jan 2004 04:36:06 -0800 -Date: Sat, 10 Jan 2004 04:36:06 -0800 -From: Kevin Brown -To: PostgreSQL-development -Subject: Re: [HACKERS] psql \d option list overloaded -Message-ID: <20040110123605.GA2608@filer> -Mail-Followup-To: Kevin Brown , - PostgreSQL-development -References: <3FEE6DFB.9040408@lorenso.com> <200401040125.i041PLR14687@candle.pha.pa.us> <20040104191322.GD8524@posixnap.net> <3FF8C4E6.9090008@lorenso.com> <20040105154534.GF8524@posixnap.net> -MIME-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -Content-Transfer-Encoding: 7bit -Content-Disposition: inline -In-Reply-To: <20040105154534.GF8524@posixnap.net> -Organization: Frobozzco International -User-Agent: Mutt/1.5.4i -X-Virus-Scanned: by amavisd-new at postgresql.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-AntiAbuse: This header was added to track abuse, please include it with any abuse report -X-AntiAbuse: Primary Hostname - noon.pghoster.com -X-AntiAbuse: Original Domain - candle.pha.pa.us -X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] -X-AntiAbuse: Sender Address Domain - postgresql.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.8 required=5.0 tests=BAYES_00,HTML_MESSAGE - autolearn=ham version=2.61 -Status: OR - -Alex J. Avriette wrote: -> On Sun, Jan 04, 2004 at 07:59:02PM -0600, D. Dante Lorenso wrote: -> -> > Anything other than simple, short commands is a waste, IMHO. I can easily -> > remember SHOW DATABASES and SHOW TABLES and DESC
, because they -> > reflect -> > my intensions directly and 'make sense'. -> -> What makes sense to me in csh doesn't make sense in a bourne shell. - -And yet, bash has !$ and job control just like csh, even though they're -not standard Bourne-shell features. - -It's not a bad thing to adopt good ideas from other projects. - -> You can't expect all applications to work correctly. - -You can't expect this anyway, at least when dealing with cross-database -applications. The intersection of the SQL feature sets across all the -major database engines is pretty limited -- small enough that you'll -almost certainly end up using something database-specific when attempting -to do anything truly nontrivial. - -> I'd like to second -> Peter's "yep" when asked if he could remember all the various \d* -> commands. It really comes down to whether you're trying. New software -> (even though you may have been using it for a year) requires some -> adjustment. - -This is true, but it's no argument against implementing "show -databases", "show tables", and "describe". - -Every database engine is different, but in the case of PG it makes sense -to adopt the best methods we can find. A consistent and easy to -remember way of showing the various entities in psql (at the very least) -would be of great advantage. It's something that MySQL gets right. As -it turns out, we already have "SHOW" in psql and it's used for something -else. So we might instead use something else (e.g. "VIEW") instead. - -Either way, a single command that takes as its argument the type of entity -you want to see would be extremely useful, and much easier to remember -than what we currently have -- because the names of the entities that -are available are already well-defined and are likely known to the user -already. - -> > What's more important is the ability to use these commands from any -> > interface not just 'psql' client. I think 'psql' already has the slash -> > commands. No need to create NEW slash commands there... -> > -> > >If you want to find out how to show the databases in sql, use psql -E. -> > > -> > > -> > Have you actually done that? OMG! -> -> Yes, I do it frequently. You may notice a recent post of mine used -> exactly that output. - -Now do it from within psql. - -It's \l, as it turns out. This violates the principle of least surprise -because psql generally uses \d* to show entities. - -> > 3) like MySQL does it... -> > -> > SHOW TABLES; -> -> Should postgres also support the '#' comment? What other non-sql -> sqlisms should we support? - -PG already has a number of PG-specific features. Adding more, -*especially* if they happen to be compatible with other databases, isn't -going to hurt much. - -No, the thing to worry about here is whether or not these commands -("SHOW", for instance) will appear in the SQL spec and will have a -completely different meaning from the meaning in PG. Also of concern is -that "SHOW" is already reserved and used for something else. We'd have -to use something other than "SHOW" for the purpose being discussed. - -> > There's something to be said about the 'SHOW'and 'DESC' sql-extensions -> > added into MySQL. Newbies can really 'get' it quickly. It's what really -> -> I would argue that these are not "sql extensions" at all. If you like, I -> can go over the source to verify this myself, but my guess is that MySQL -> is doing exactly what postgres is doing, and evaluating this as a macro. - -No, they are built into MySQL's backend parser. You can easily verify -this by executing these commands from within Perl or Python. They -return a table just like any other SQL command that returns data. - -> Furthermore, databases are not designed for "newbies" to jump right in -> with both feet. They are designed to be robust and stable. - -Now this is ludicrous. Yes, they're designed to be robust and stable, -but that has absolutely nothing to do with how easy they are to use. - -> Additionally, -> some SQL compliance is nice. After that, you work on features. - -If we were talking about something that went against the SQL standard -then I would agree with you. But we're talking about something that, -as far as I know, isn't in the SQL standard at all. Implementing it -won't make us noncompliant with the SQL standard any more than the -implementation of CREATE INDEX has. - -> Changing the interface so that you or others don't have to read the -> documentation smacks of laziness. - -Really? One could make the same argument for standards of any kind, -yes? :-) - -> Somebody like Bruce, Peter, or Tom (or indeed somebody else) is going -> to waste yet more time making things like this available to somebody -> who probably won't read any of the other documentation either, and will -> wind up on irc pestering somebody like myself, Dave, or Neil. Why is -> this progress? - -It's progress because it will keep those people from pestering someone -in the know about how to show the available databases, or how to -describe a table. - -> > sold me on MySQL when I first learned it. For me, it's like: -> > -> > 'dir' in DOS, -> > 'ls' in Unix -> > 'SHOW' in MySQL -> > ??? in PostgreSQL ? -> -> We've been over this. It's \d*. - -For listing databases it's \l. Not exactly consistent with the rest of -the related psql commands. - -> > Sure, with time as my database needs grew and I matured as a developer, -> > I eventually gained more respect for PostgreSQL and have made the switch -> > even without this feature, but to this day, I really think MySQL *did it -> > right* with those extensions. You can't become a PostgreSQL guru without -> > being a newbie first. I vote we make it easier for newbies. -> -> What really frightens me here is that I know of several applications (shudder, -> "LAMP" applications) which use the output of "show tables" or other of your -> "extensions." The problem with this is precisely that it /isn't/ sql, and it -> can't be supported as a static command. - -Of course not. But applications which rely on information such as that -provided by "show tables" will typically not be possible to write while -adhering to the feature intersection of all major databases anyway. - -> It is intended to be there for people -> to use interactively. - -Nonsense. It's there to be used. Whether it's used interactively or -not is irrelevant. The command provides useful information. But see -below. - -> Making "pseudo sql" will encourage more developers to -> (and I'd apologize for this if it weren't true) code in Postgres the same -> lazy way they code in MySQL. - -This is a strawman argument, although I understand your concern here. -To be honest, for application development I'd much rather see people -use information_schema, but that's only because information_schema is -in the SQL standard and as such should be the preferred way to retrieve -the information that the "SHOW" commands in MySQL return. That said, -the inclusion of information_schema is a very recent development on the -PostgreSQL side of things, and doesn't even exist on some other major -databases such as MSSQL. - -Of course, a PG equivalent to MySQL's "show" would be an even more -recent development... :-) - - --- -Kevin Brown kevin@sysexperts.com - ----------------------------(end of broadcast)--------------------------- -TIP 9: the planner will ignore your desire to choose an index scan if your - joining column's datatypes do not match - -From pgsql-hackers-owner+M48947@postgresql.org Sat Jan 10 08:12:39 2004 -Return-path: -Received: from hosting.commandprompt.com (216.commandprompt.com [207.173.200.216]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i0ADCbX21504 - for ; Sat, 10 Jan 2004 08:12:38 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by hosting.commandprompt.com (8.11.6/8.11.6) with ESMTP id i0AD8Gk19951; - Sat, 10 Jan 2004 05:09:17 -0800 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (neptune.hub.org [200.46.204.2]) - by svr1.postgresql.org (Postfix) with ESMTP id 7758BD1D54C - for ; Sat, 10 Jan 2004 13:08:00 +0000 (GMT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) - with ESMTP id 02466-03 - for ; - Sat, 10 Jan 2004 09:07:32 -0400 (AST) -Received: from curie.credativ.org (credativ.com [217.160.209.18]) - by svr1.postgresql.org (Postfix) with ESMTP id 64975D1D54E - for ; Sat, 10 Jan 2004 09:07:28 -0400 (AST) -Received: from localhost (localhost [127.0.0.1]) - by curie.credativ.org (Postfix) with ESMTP - id 765E356243; Sat, 10 Jan 2004 14:07:25 +0100 (CET) -Received: from colt.pezone.net (dsl-213-023-254-001.arcor-ip.net [213.23.254.1]) - (using TLSv1 with cipher RC4-MD5 (128/128 bits)) - (No client certificate requested) - by curie.credativ.org (Postfix) with ESMTP - id B98A65623C; Sat, 10 Jan 2004 14:07:24 +0100 (CET) -From: Peter Eisentraut -To: Kevin Brown , - PostgreSQL-development -Subject: Re: [HACKERS] psql \d option list overloaded -Date: Sat, 10 Jan 2004 14:07:24 +0100 -User-Agent: KMail/1.5.1 -References: <3FEE6DFB.9040408@lorenso.com> <20040105154534.GF8524@posixnap.net> <20040110123605.GA2608@filer> -In-Reply-To: <20040110123605.GA2608@filer> -MIME-Version: 1.0 -Content-Type: text/plain; - charset="iso-8859-1" -Content-Transfer-Encoding: 7bit -Content-Disposition: inline -Message-ID: <200401101407.24766.peter_e@gmx.net> -X-Virus-Scanned: by AMaViS at credativ.com -X-Virus-Scanned: by amavisd-new at postgresql.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham - version=2.61 -Status: OR - -Kevin Brown wrote: -> Every database engine is different, but in the case of PG it makes -> sense to adopt the best methods we can find. A consistent and easy -> to remember way of showing the various entities in psql (at the very -> least) would be of great advantage. It's something that MySQL gets -> right. As it turns out, we already have "SHOW" in psql and it's used -> for something else. So we might instead use something else (e.g. -> "VIEW") instead. - -What is wrong with - -SELECT * FROM information_schema.tables; - -? If it's too much to type, put information_schema in the path. This -syntax has the advantage that you can use qualifications and other SQL -features. And you can build customized views on top of it. Does SHOW -TABLES or whatever it might be called support that? - - ----------------------------(end of broadcast)--------------------------- -TIP 9: the planner will ignore your desire to choose an index scan if your - joining column's datatypes do not match - -From pgsql-hackers-owner+M48948@postgresql.org Sat Jan 10 11:30:24 2004 -Return-path: -Received: from noon.pghoster.com (noon.pghoster.com [64.246.0.64]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i0AGUMX12749 - for ; Sat, 10 Jan 2004 11:30:23 -0500 (EST) -Received: from svr1.postgresql.org ([200.46.204.71] helo=postgresql.org) - by noon.pghoster.com with esmtp (Exim 4.24) - id 1AfLzP-0000hA-JY; Sat, 10 Jan 2004 10:29:31 -0600 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (neptune.hub.org [200.46.204.2]) - by svr1.postgresql.org (Postfix) with ESMTP id A69DED1B51D - for ; Sat, 10 Jan 2004 16:28:00 +0000 (GMT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) - with ESMTP id 14752-08 - for ; - Sat, 10 Jan 2004 12:27:29 -0400 (AST) -Received: from zigo.dhs.org (as2-4-3.an.g.bonet.se [194.236.34.191]) - by svr1.postgresql.org (Postfix) with ESMTP id 89156D1D560 - for ; Sat, 10 Jan 2004 12:27:23 -0400 (AST) -Received: from zigo.zigo.dhs.org (zigo.zigo.dhs.org [192.168.0.1]) - by zigo.dhs.org (Postfix) with ESMTP - id 9D7EF8E0D; Sat, 10 Jan 2004 17:27:21 +0100 (CET) -Date: Sat, 10 Jan 2004 17:27:21 +0100 (CET) -From: =?ISO-8859-1?Q?Dennis_Bj=F6rklund?= -To: Peter Eisentraut -cc: Kevin Brown , - PostgreSQL-development -Subject: Re: [HACKERS] psql \d option list overloaded -In-Reply-To: <200401101407.24766.peter_e@gmx.net> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=ISO-8859-1 -Content-Transfer-Encoding: 8BIT -X-Virus-Scanned: by amavisd-new at postgresql.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-AntiAbuse: This header was added to track abuse, please include it with any abuse report -X-AntiAbuse: Primary Hostname - noon.pghoster.com -X-AntiAbuse: Original Domain - candle.pha.pa.us -X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] -X-AntiAbuse: Sender Address Domain - postgresql.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham - version=2.61 -Status: OR - -On Sat, 10 Jan 2004, Peter Eisentraut wrote: - -> > to remember way of showing the various entities in psql (at the very -> > least) would be of great advantage. It's something that MySQL gets -> > right. As it turns out, we already have "SHOW" in psql and it's used -> > for something else. -> -> What is wrong with -> -> SELECT * FROM information_schema.tables; - -The result is very hard to read since it's so much of it (try column -instead of tables). The \xx commands do some nice formatting you don't -get from the above. - -I would rather have long commands so one can write - -\describe_table foo - -and have the tab completion work for these of course (only for the long -commands, the \dt and such does not belong in completion). - -The information schema is nice, but it's not what I want to use at the -prompt to view the content of the database. - --- -/Dennis Björklund - - ----------------------------(end of broadcast)--------------------------- -TIP 2: you can get off all lists at once with the unregister command - (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) - -From pgsql-hackers-owner+M48950@postgresql.org Sat Jan 10 13:30:54 2004 -Return-path: -Received: from noon.pghoster.com (noon.pghoster.com [64.246.0.64]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i0AIUrX25980 - for ; Sat, 10 Jan 2004 13:30:53 -0500 (EST) -Received: from svr1.postgresql.org ([200.46.204.71] helo=postgresql.org) - by noon.pghoster.com with esmtp (Exim 4.24) - id 1AfNsG-0002v6-Lh; Sat, 10 Jan 2004 12:30:17 -0600 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (neptune.hub.org [200.46.204.2]) - by svr1.postgresql.org (Postfix) with ESMTP id 92952D1D56E - for ; Sat, 10 Jan 2004 18:28:45 +0000 (GMT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) - with ESMTP id 33571-06 - for ; - Sat, 10 Jan 2004 14:28:14 -0400 (AST) -Received: from smtp.istop.com (dci.doncaster.on.ca [66.11.168.194]) - by svr1.postgresql.org (Postfix) with ESMTP id 9FBA4D1D572 - for ; Sat, 10 Jan 2004 14:28:13 -0400 (AST) -Received: from stark.xeocode.com (gsstark.mtl.istop.com [66.11.160.162]) - by smtp.istop.com (Postfix) with ESMTP - id 1CC0837658; Sat, 10 Jan 2004 13:28:09 -0500 (EST) -Received: from localhost - ([127.0.0.1] helo=stark.xeocode.com ident=foobar) - by stark.xeocode.com with smtp (Exim 3.36 #1 (Debian)) - id 1AfNqD-0001y1-00; Sat, 10 Jan 2004 13:28:09 -0500 -To: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] psql \d option list overloaded -References: -In-Reply-To: -From: Greg Stark -Organization: The Emacs Conspiracy; member since 1992 -Date: 10 Jan 2004 13:28:08 -0500 -Message-ID: <878ykf4q13.fsf@stark.xeocode.com> -Lines: 22 -User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3 -MIME-Version: 1.0 -Content-Type: text/plain; charset=iso-8859-1 -Content-Transfer-Encoding: 8bit -X-Virus-Scanned: by amavisd-new at postgresql.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-AntiAbuse: This header was added to track abuse, please include it with any abuse report -X-AntiAbuse: Primary Hostname - noon.pghoster.com -X-AntiAbuse: Original Domain - candle.pha.pa.us -X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] -X-AntiAbuse: Sender Address Domain - postgresql.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham - version=2.61 -Status: OR - - -Dennis Björklund writes: - -> I would rather have long commands so one can write -> -> \describe_table foo - -I would think it would be better to keep everything under a single command and -have a 1-1 correspondence to \d. Ie, just add a long form syntax following the -existing \d. \d would become just an obvious set of abbreviations. - -So for example: - -\describe table foo => \dt foo -\describe index foo => \di foo -\describe aggregate foo => \da foo -\describe operator foo => \do foo - -... - --- -greg - - ----------------------------(end of broadcast)--------------------------- -TIP 6: Have you searched our list archives? - - http://archives.postgresql.org - -From pgsql-hackers-owner+M48952@postgresql.org Sat Jan 10 14:17:26 2004 -Return-path: -Received: from noon.pghoster.com (noon.pghoster.com [64.246.0.64]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i0AJHOX29152 - for ; Sat, 10 Jan 2004 14:17:25 -0500 (EST) -Received: from svr1.postgresql.org ([200.46.204.71] helo=postgresql.org) - by noon.pghoster.com with esmtp (Exim 4.24) - id 1AfObD-0003wE-9N; Sat, 10 Jan 2004 13:16:43 -0600 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (neptune.hub.org [200.46.204.2]) - by svr1.postgresql.org (Postfix) with ESMTP id 27045D1B498 - for ; Sat, 10 Jan 2004 19:15:51 +0000 (GMT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) - with ESMTP id 42506-02 - for ; - Sat, 10 Jan 2004 15:15:21 -0400 (AST) -Received: from sss.pgh.pa.us (unknown [192.204.191.242]) - by svr1.postgresql.org (Postfix) with ESMTP id 143C4D1B4C5 - for ; Sat, 10 Jan 2004 15:15:20 -0400 (AST) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.12.10/8.12.10) with ESMTP id i0AJFJ19011695; - Sat, 10 Jan 2004 14:15:19 -0500 (EST) -To: Greg Stark -cc: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] psql \d option list overloaded -In-Reply-To: <878ykf4q13.fsf@stark.xeocode.com> -References: <878ykf4q13.fsf@stark.xeocode.com> -Comments: In-reply-to Greg Stark - message dated "10 Jan 2004 13:28:08 -0500" -Date: Sat, 10 Jan 2004 14:15:19 -0500 -Message-ID: <11694.1073762119@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at postgresql.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-AntiAbuse: This header was added to track abuse, please include it with any abuse report -X-AntiAbuse: Primary Hostname - noon.pghoster.com -X-AntiAbuse: Original Domain - candle.pha.pa.us -X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] -X-AntiAbuse: Sender Address Domain - postgresql.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham - version=2.61 -Status: OR - -Greg Stark writes: -> So for example: - -> \describe table foo => \dt foo -> \describe index foo => \di foo -> \describe aggregate foo => \da foo -> \describe operator foo => \do foo - -It doesn't seem to me that this buys much except verboseness, though. - -ISTM there are three fundamental problems with \d and friends: - - 1. Some people have a hard time remembering the commands. - 2. Some people aren't using psql. - 3. psql keeps breaking across backend versions because the - needed commands change. - -I don't see a lot of value in addressing just one of these problem -areas, when we could instead do something that addresses all three. - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 9: the planner will ignore your desire to choose an index scan if your - joining column's datatypes do not match - -From pgsql-hackers-owner+M48954@postgresql.org Sat Jan 10 19:19:30 2004 -Return-path: -Received: from noon.pghoster.com (noon.pghoster.com [64.246.0.64]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i0B0JPX16197 - for ; Sat, 10 Jan 2004 19:19:29 -0500 (EST) -Received: from svr1.postgresql.org ([200.46.204.71] helo=postgresql.org) - by noon.pghoster.com with esmtp (Exim 4.24) - id 1AfTJQ-0000BC-Jg; Sat, 10 Jan 2004 18:18:40 -0600 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (neptune.hub.org [200.46.204.2]) - by svr1.postgresql.org (Postfix) with ESMTP id 17450D1B465 - for ; Sun, 11 Jan 2004 00:17:39 +0000 (GMT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) - with ESMTP id 71332-04 - for ; - Sat, 10 Jan 2004 20:17:10 -0400 (AST) -Received: from lakemtao03.cox.net (lakemtao03.cox.net [68.1.17.242]) - by svr1.postgresql.org (Postfix) with ESMTP id 937E9D1B461 - for ; Sat, 10 Jan 2004 20:17:06 -0400 (AST) -Received: from louche.swelter.net ([68.98.183.165]) by lakemtao03.cox.net - (InterMail vM.5.01.06.05 201-253-122-130-105-20030824) with ESMTP - id <20040111001710.UTGO2192.lakemtao03.cox.net@louche.swelter.net> - for ; - Sat, 10 Jan 2004 19:17:10 -0500 -Received: by louche.swelter.net (Postfix, from userid 513) - id 0B28AB; Sat, 10 Jan 2004 19:16:59 -0500 (EST) -Received: from localhost (louche.swelter.net [127.0.0.1]) - by louche.swelter.net (Postfix) with ESMTP id 22CDBB - for ; Sat, 10 Jan 2004 19:16:58 -0500 (EST) -Date: Sun, 11 Jan 2004 00:16:58 +0000 (UTC) -From: Jon Jensen -X-X-Sender: jon@louche.swelter.net -To: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] psql \d option list overloaded -In-Reply-To: <11694.1073762119@sss.pgh.pa.us> -Message-ID: -References: - <878ykf4q13.fsf@stark.xeocode.com> <11694.1073762119@sss.pgh.pa.us> -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -X-Virus-Scanned: by amavisd-new at postgresql.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-AntiAbuse: This header was added to track abuse, please include it with any abuse report -X-AntiAbuse: Primary Hostname - noon.pghoster.com -X-AntiAbuse: Original Domain - candle.pha.pa.us -X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] -X-AntiAbuse: Sender Address Domain - postgresql.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham - version=2.61 -Status: OR - -On Sat, 10 Jan 2004, Tom Lane wrote: - -> ISTM there are three fundamental problems with \d and friends: -> -> 1. Some people have a hard time remembering the commands. -> 2. Some people aren't using psql. -> 3. psql keeps breaking across backend versions because the -> needed commands change. -> -> I don't see a lot of value in addressing just one of these problem -> areas, when we could instead do something that addresses all three. - -I agree, at least for #2 and #3. But I just don't understand #1. Anything -is hard to remember when you're just starting to learn it. But it's still -faster to type \? then \dt than it is to type "show tables". And -"show tables" is hard (relatively speaking) for me to remember because I'm -used to psql's way of doing things, since I mostly use it. - -Jon - ----------------------------(end of broadcast)--------------------------- -TIP 7: don't forget to increase your free space map settings - -From pgsql-hackers-owner+M48955@postgresql.org Sat Jan 10 20:09:33 2004 -Return-path: -Received: from noon.pghoster.com (noon.pghoster.com [64.246.0.64]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i0B19WX22168 - for ; Sat, 10 Jan 2004 20:09:33 -0500 (EST) -Received: from svr1.postgresql.org ([200.46.204.71] helo=postgresql.org) - by noon.pghoster.com with esmtp (Exim 4.24) - id 1AfU65-000194-6w; Sat, 10 Jan 2004 19:08:57 -0600 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (neptune.hub.org [200.46.204.2]) - by svr1.postgresql.org (Postfix) with ESMTP id B6F12D1B528 - for ; Sun, 11 Jan 2004 01:07:56 +0000 (GMT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) - with ESMTP id 70575-10 - for ; - Sat, 10 Jan 2004 21:07:28 -0400 (AST) -Received: from curie.credativ.org (credativ.com [217.160.209.18]) - by svr1.postgresql.org (Postfix) with ESMTP id EA5B5D1B48B - for ; Sat, 10 Jan 2004 21:07:24 -0400 (AST) -Received: from localhost (localhost [127.0.0.1]) - by curie.credativ.org (Postfix) with ESMTP - id DD86C56243; Sun, 11 Jan 2004 02:07:26 +0100 (CET) -Received: from colt.pezone.net (dsl-213-023-254-001.arcor-ip.net [213.23.254.1]) - (using TLSv1 with cipher RC4-MD5 (128/128 bits)) - (No client certificate requested) - by curie.credativ.org (Postfix) with ESMTP - id EA9C35623C; Sun, 11 Jan 2004 02:07:25 +0100 (CET) -From: Peter Eisentraut -To: Tom Lane , Greg Stark -Subject: Re: [HACKERS] psql \d option list overloaded -Date: Sun, 11 Jan 2004 02:07:25 +0100 -User-Agent: KMail/1.5.1 -cc: pgsql-hackers@postgresql.org -References: <878ykf4q13.fsf@stark.xeocode.com> <11694.1073762119@sss.pgh.pa.us> -In-Reply-To: <11694.1073762119@sss.pgh.pa.us> -MIME-Version: 1.0 -Content-Type: text/plain; - charset="iso-8859-15" -Content-Transfer-Encoding: 7bit -Content-Disposition: inline -Message-ID: <200401110207.26299.peter_e@gmx.net> -X-Virus-Scanned: by AMaViS at credativ.com -X-Virus-Scanned: by amavisd-new at postgresql.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-AntiAbuse: This header was added to track abuse, please include it with any abuse report -X-AntiAbuse: Primary Hostname - noon.pghoster.com -X-AntiAbuse: Original Domain - candle.pha.pa.us -X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] -X-AntiAbuse: Sender Address Domain - postgresql.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham - version=2.61 -Status: OR - -Tom Lane wrote: -> 2. Some people aren't using psql. - -I don't see why this is an issue. People not using psql are either -using a GUI, which presumably supports plenty of "show" and "describe" -functionality, or they're writing their own program, in which case it -doesn't really matter how short or easy to remember the commands are. - - ----------------------------(end of broadcast)--------------------------- -TIP 2: you can get off all lists at once with the unregister command - (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) - -From pgsql-hackers-owner+M48956@postgresql.org Sat Jan 10 20:19:59 2004 -Return-path: -Received: from hosting.commandprompt.com (216.commandprompt.com [207.173.200.216]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i0B1JoX23197 - for ; Sat, 10 Jan 2004 20:19:56 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by hosting.commandprompt.com (8.11.6/8.11.6) with ESMTP id i0B1E2k19636; - Sat, 10 Jan 2004 17:15:07 -0800 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (neptune.hub.org [200.46.204.2]) - by svr1.postgresql.org (Postfix) with ESMTP id 34FE7D1B436 - for ; Sun, 11 Jan 2004 01:13:46 +0000 (GMT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) - with ESMTP id 76096-05 - for ; - Sat, 10 Jan 2004 21:13:18 -0400 (AST) -Received: from curie.credativ.org (credativ.com [217.160.209.18]) - by svr1.postgresql.org (Postfix) with ESMTP id 56B90D1B430 - for ; Sat, 10 Jan 2004 21:13:15 -0400 (AST) -Received: from localhost (localhost [127.0.0.1]) - by curie.credativ.org (Postfix) with ESMTP - id 91B2A56243; Sun, 11 Jan 2004 02:13:18 +0100 (CET) -Received: from colt.pezone.net (dsl-213-023-254-001.arcor-ip.net [213.23.254.1]) - (using TLSv1 with cipher RC4-MD5 (128/128 bits)) - (No client certificate requested) - by curie.credativ.org (Postfix) with ESMTP - id 7D86F5623C; Sun, 11 Jan 2004 02:13:17 +0100 (CET) -From: Peter Eisentraut -To: Dennis =?iso-8859-1?q?Bj=F6rklund?= -Subject: Re: [HACKERS] psql \d option list overloaded -Date: Sun, 11 Jan 2004 02:13:17 +0100 -User-Agent: KMail/1.5.1 -cc: Kevin Brown , - PostgreSQL-development -References: -In-Reply-To: -MIME-Version: 1.0 -Content-Type: text/plain; - charset="iso-8859-1" -Content-Transfer-Encoding: 8bit -Content-Disposition: inline -Message-ID: <200401110213.17581.peter_e@gmx.net> -X-Virus-Scanned: by AMaViS at credativ.com -X-Virus-Scanned: by amavisd-new at postgresql.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham - version=2.61 -Status: OR - -Dennis Björklund wrote: -> > What is wrong with -> > -> > SELECT * FROM information_schema.tables; -> -> The result is very hard to read since it's so much of it (try column -> instead of tables). The \xx commands do some nice formatting you -> don't get from the above. - -This is an interesting point to remember for those that are advocating -pushing psql's queries into the backend. psql's queries are optimized -for monospaced text screens of limited size. Unless someone else is -writing a command-line client, there would be little reuse effect, -because any given application will have different display requirements. - -(Another problem with pushing psql's queries into the backend is that -much of the output that psql makes is not a single table. Sometimes -there is more than one table, or the information is in the table -footers. It'd be quite complicated to make the backend produce those -kinds of displays.) - - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M48957@postgresql.org Sun Jan 11 04:34:27 2004 -Return-path: -Received: from hosting.commandprompt.com (216.commandprompt.com [207.173.200.216]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i0B9YPX11678 - for ; Sun, 11 Jan 2004 04:34:26 -0500 (EST) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by hosting.commandprompt.com (8.11.6/8.11.6) with ESMTP id i0B9U9k04632; - Sun, 11 Jan 2004 01:31:12 -0800 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (neptune.hub.org [200.46.204.2]) - by svr1.postgresql.org (Postfix) with ESMTP id C49AFD1D555 - for ; Sun, 11 Jan 2004 09:29:54 +0000 (GMT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) - with ESMTP id 26981-08 - for ; - Sun, 11 Jan 2004 05:29:25 -0400 (AST) -Received: from zigo.dhs.org (as2-4-3.an.g.bonet.se [194.236.34.191]) - by svr1.postgresql.org (Postfix) with ESMTP id A75DDD1D27D - for ; Sun, 11 Jan 2004 05:29:22 -0400 (AST) -Received: from zigo.zigo.dhs.org (zigo.zigo.dhs.org [192.168.0.1]) - by zigo.dhs.org (Postfix) with ESMTP - id B8DED8E0D; Sun, 11 Jan 2004 10:29:23 +0100 (CET) -Date: Sun, 11 Jan 2004 10:29:23 +0100 (CET) -From: Dennis Bjorklund -To: Peter Eisentraut -cc: Kevin Brown , - PostgreSQL-development -Subject: Re: [HACKERS] psql \d option list overloaded -In-Reply-To: <200401110213.17581.peter_e@gmx.net> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=ISO-8859-1 -Content-Transfer-Encoding: 8BIT -X-Virus-Scanned: by amavisd-new at postgresql.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham - version=2.61 -Status: OR - -On Sun, 11 Jan 2004, Peter Eisentraut wrote: - -> Another problem with pushing psql's queries into the backend is that -> much of the output that psql makes is not a single table. Sometimes -> there is more than one table, or the information is in the table -> footers. - -Yes, pushing the \xx commands into the server makes no sense to me at all. - -The commands in psql are very specific for psql. I don't see why you ever -want to do SHOW TABLES except at the command line in psql. If your -application wants to find all tables in the database, then we have the -standard sql way, which is the information schema. - -The argument that "show tables" is easier to remember then \dt might be -true, but to me that just means that we should make psql better by adding -\describe_table and such, not to push psql code into the server. - -Making a couple of views that are pg specific to make it easier to get -information out could be good however. The information schema does not -always contain all information one might want. Making specialised SQL -commands for it I'm not in favor of at all. - --- -/Dennis Björklund - - ----------------------------(end of broadcast)--------------------------- -TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org - -From pgsql-hackers-owner+M48965@postgresql.org Sun Jan 11 12:20:30 2004 -Return-path: -Received: from noon.pghoster.com (noon.pghoster.com [64.246.0.64]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i0BHKTX05825 - for ; Sun, 11 Jan 2004 12:20:29 -0500 (EST) -Received: from svr1.postgresql.org ([200.46.204.71] helo=postgresql.org) - by noon.pghoster.com with esmtp (Exim 4.24) - id 1AfjFg-00073z-AN; Sun, 11 Jan 2004 11:19:52 -0600 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (neptune.hub.org [200.46.204.2]) - by svr1.postgresql.org (Postfix) with ESMTP id 46DAED1B4AD - for ; Sun, 11 Jan 2004 17:18:57 +0000 (GMT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) - with ESMTP id 77771-03 - for ; - Sun, 11 Jan 2004 13:18:26 -0400 (AST) -Received: from sss.pgh.pa.us (unknown [192.204.191.242]) - by svr1.postgresql.org (Postfix) with ESMTP id 4F237D1B51D - for ; Sun, 11 Jan 2004 13:18:25 -0400 (AST) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.12.10/8.12.10) with ESMTP id i0BHI519016986; - Sun, 11 Jan 2004 12:18:05 -0500 (EST) -To: Peter Eisentraut -cc: Greg Stark , pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] psql \d option list overloaded -In-Reply-To: <200401110207.26299.peter_e@gmx.net> -References: <878ykf4q13.fsf@stark.xeocode.com> <11694.1073762119@sss.pgh.pa.us> <200401110207.26299.peter_e@gmx.net> -Comments: In-reply-to Peter Eisentraut - message dated "Sun, 11 Jan 2004 02:07:25 +0100" -Date: Sun, 11 Jan 2004 12:18:05 -0500 -Message-ID: <16985.1073841485@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at postgresql.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-AntiAbuse: This header was added to track abuse, please include it with any abuse report -X-AntiAbuse: Primary Hostname - noon.pghoster.com -X-AntiAbuse: Original Domain - candle.pha.pa.us -X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] -X-AntiAbuse: Sender Address Domain - postgresql.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham - version=2.61 -Status: OR - -Peter Eisentraut writes: -> Tom Lane wrote: ->> 2. Some people aren't using psql. - -> I don't see why this is an issue. People not using psql are either -> using a GUI, which presumably supports plenty of "show" and "describe" -> functionality, or they're writing their own program, in which case it -> doesn't really matter how short or easy to remember the commands are. - -But this interacts with point 3 (psql breaks on every new backend -version). It's not desirable to have every GUI and large custom program -implementing its own set of metadata inquiry commands: they all have -to go through the same update pain as psql. Perhaps if people start to -rely on information_schema for those things, life will get better, -but I'm unconvinced that will happen. psql itself certainly hasn't -moved in that direction. - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 8: explain analyze is your friend - -From pgsql-hackers-owner+M48981@postgresql.org Sun Jan 11 20:52:09 2004 -Return-path: -Received: from noon.pghoster.com (noon.pghoster.com [64.246.0.64]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i0C1q8X05166 - for ; Sun, 11 Jan 2004 20:52:09 -0500 (EST) -Received: from svr1.postgresql.org ([200.46.204.71] helo=postgresql.org) - by noon.pghoster.com with esmtp (Exim 4.24) - id 1AfrF5-0005Pj-Aq; Sun, 11 Jan 2004 19:51:47 -0600 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (neptune.hub.org [200.46.204.2]) - by svr1.postgresql.org (Postfix) with ESMTP id 654B2D1B519 - for ; Mon, 12 Jan 2004 01:50:40 +0000 (GMT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) - with ESMTP id 57929-03 - for ; - Sun, 11 Jan 2004 21:50:12 -0400 (AST) -Received: from lakemtao04.cox.net (lakemtao04.cox.net [68.1.17.241]) - by svr1.postgresql.org (Postfix) with ESMTP id DD464D1D576 - for ; Sun, 11 Jan 2004 21:50:07 -0400 (AST) -Received: from [192.168.0.13] ([68.105.168.121]) by lakemtao04.cox.net - (InterMail vM.5.01.06.05 201-253-122-130-105-20030824) with ESMTP - id <20040112015012.MDVR19895.lakemtao04.cox.net@[192.168.0.13]>; - Sun, 11 Jan 2004 20:50:12 -0500 -From: Robert Treat -To: Jon Jensen , pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] psql \d option list overloaded -Date: Sun, 11 Jan 2004 20:50:08 -0500 -User-Agent: KMail/1.5 -References: <11694.1073762119@sss.pgh.pa.us> -In-Reply-To: -MIME-Version: 1.0 -Content-Type: text/plain; - charset="iso-8859-1" -Content-Transfer-Encoding: 7bit -Content-Disposition: inline -Message-ID: <200401112050.09142.xzilla@users.sourceforge.net> -X-Virus-Scanned: by amavisd-new at postgresql.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-AntiAbuse: This header was added to track abuse, please include it with any abuse report -X-AntiAbuse: Primary Hostname - noon.pghoster.com -X-AntiAbuse: Original Domain - candle.pha.pa.us -X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] -X-AntiAbuse: Sender Address Domain - postgresql.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham - version=2.61 -Status: OR - -On Saturday 10 January 2004 19:16, Jon Jensen wrote: -> On Sat, 10 Jan 2004, Tom Lane wrote: -> > ISTM there are three fundamental problems with \d and friends: -> > -> > 1. Some people have a hard time remembering the commands. -> > 2. Some people aren't using psql. -> > 3. psql keeps breaking across backend versions because the -> > needed commands change. -> > -> > I don't see a lot of value in addressing just one of these problem -> > areas, when we could instead do something that addresses all three. -> -> I agree, at least for #2 and #3. But I just don't understand #1. Anything -> is hard to remember when you're just starting to learn it. But it's still -> faster to type \? then \dt than it is to type "show tables". And -> "show tables" is hard (relatively speaking) for me to remember because I'm -> used to psql's way of doing things, since I mostly use it. -> - -I'd second this point; I've certainly stumbled over the "show" syntax when -trying to get anything other than tables in mysql. - -Robert Treat --- -Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL - ----------------------------(end of broadcast)--------------------------- -TIP 2: you can get off all lists at once with the unregister command - (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) - -From pgsql-hackers-owner+M48993@postgresql.org Mon Jan 12 12:50:40 2004 -Return-path: -Received: from noon.pghoster.com (noon.pghoster.com [64.246.0.64]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i0CHocX04973 - for ; Mon, 12 Jan 2004 12:50:39 -0500 (EST) -Received: from svr1.postgresql.org ([200.46.204.71] helo=postgresql.org) - by noon.pghoster.com with esmtp (Exim 4.24) - id 1Ag6Ck-0008V2-E5; Mon, 12 Jan 2004 11:50:22 -0600 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (neptune.hub.org [200.46.204.2]) - by svr1.postgresql.org (Postfix) with ESMTP id ED886D1D17D - for ; Mon, 12 Jan 2004 17:49:16 +0000 (GMT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) - with ESMTP id 17616-05 - for ; - Mon, 12 Jan 2004 13:48:46 -0400 (AST) -Received: from curie.credativ.org (credativ.com [217.160.209.18]) - by svr1.postgresql.org (Postfix) with ESMTP id 09691D1B467 - for ; Mon, 12 Jan 2004 13:48:45 -0400 (AST) -Received: from localhost (localhost [127.0.0.1]) - by curie.credativ.org (Postfix) with ESMTP - id 45CA05626B; Mon, 12 Jan 2004 18:48:44 +0100 (CET) -Received: from colt.pezone.net (dsl-213-023-254-001.arcor-ip.net [213.23.254.1]) - (using TLSv1 with cipher RC4-MD5 (128/128 bits)) - (No client certificate requested) - by curie.credativ.org (Postfix) with ESMTP - id 2F90D5623C; Mon, 12 Jan 2004 18:48:43 +0100 (CET) -From: Peter Eisentraut -To: Tom Lane -Subject: Re: [HACKERS] psql \d option list overloaded -Date: Mon, 12 Jan 2004 18:48:43 +0100 -User-Agent: KMail/1.5.1 -cc: Greg Stark , pgsql-hackers@postgresql.org -References: <200401110207.26299.peter_e@gmx.net> <16985.1073841485@sss.pgh.pa.us> -In-Reply-To: <16985.1073841485@sss.pgh.pa.us> -MIME-Version: 1.0 -Content-Type: text/plain; - charset="iso-8859-15" -Content-Transfer-Encoding: 7bit -Content-Disposition: inline -Message-ID: <200401121848.43765.peter_e@gmx.net> -X-Virus-Scanned: by AMaViS at credativ.com -X-Virus-Scanned: by amavisd-new at postgresql.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-AntiAbuse: This header was added to track abuse, please include it with any abuse report -X-AntiAbuse: Primary Hostname - noon.pghoster.com -X-AntiAbuse: Original Domain - candle.pha.pa.us -X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] -X-AntiAbuse: Sender Address Domain - postgresql.org -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham - version=2.61 -Status: OR - -Tom Lane wrote: -> But this interacts with point 3 (psql breaks on every new backend -> version). It's not desirable to have every GUI and large custom -> program implementing its own set of metadata inquiry commands: they -> all have to go through the same update pain as psql. Perhaps if -> people start to rely on information_schema for those things, life -> will get better, but I'm unconvinced that will happen. psql itself -> certainly hasn't moved in that direction. - -IIRC, the two killers in psql compatibility have been outer joins and -schemas. I don't see how we could have avoided that, except with -highly specialized and static (parameter-less) commands. There have -been additional minor issues, but I suppose we could have avoided those -if we had cared to do so at all. - -Several people have in the past proposed to keep psql backward -compatible, even if only by means of - -if (version =x) { - ... -} -else if (version = y) { - ... -} - -(which would be fine by me), but apparently no one has felt pressed -enough yet. - - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - diff --git a/doc/TODO.detail/thread b/doc/TODO.detail/thread deleted file mode 100644 index 794f3d1b3e..0000000000 --- a/doc/TODO.detail/thread +++ /dev/null @@ -1,4500 +0,0 @@ -From mscott@sacadia.com Wed Nov 15 14:50:19 2000 -Received: from goldengate.kojoworldwide.com. ([216.133.4.130]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id OAA11583 - for ; Wed, 15 Nov 2000 14:50:13 -0500 (EST) -Received: from localhost (localhost [127.0.0.1]) - by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id LAA09998; - Wed, 15 Nov 2000 11:35:33 -0800 (PST) -Date: Wed, 15 Nov 2000 11:35:33 -0800 (PST) -From: Myron Scott -X-Sender: mscott@goldengate.kojoworldwide.com. -To: "Mikheev, Vadim" , - Bruce Momjian , Tom Lane -Subject: Please help with some advice -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Status: ORr - -Dear Sirs, - -I have been lurking on the PostgreSQL hackers list for about 3 months now -and your names comes up more than any with helpful info about the project -so I was hoping you could help me. - -Let me cut to the chase. I have been experimenting with 7.0.2 source to -see if I could create a mutlti-threaded version of the backend so -I could link directly from java ( I have a fe<->be protocol that I use for -my apps). Needless to say I got into much more than I bargained for. I -now have a version that works and it has some nice benefits that are very -helpful to a project that I am working on. What I gained was - -prepared statements outside of spi -batched commits (fsync) -one connection per thread - multiple threads per process - multiple processes per installation - -I never really intended for anyone else to see the work so I drifted -pretty far from the original code. I also ended up using Solaris threads -rather than pthreads, I did my own implementation of the bufmgr.c and -gram.y, and used Solaris implementation of mutex in place of S_LOCK and -TAS. I grabbed all global variables and put them in an environment -variable that is thread local. I also did some really stupid -things like making TransactionId uint64 and making all my inserts use the -same oid. - -My question is this. I would like to get some critical feedback and -suggestions about the work from others. What is the best way to go about -this? I thought about trying to create a project on greatbridge.org -but I am rather new to open source and the code needs commented properly -and cleaned up before too many try and look at it. - -Any suggestions would be greatly appreciated. - - -Thanks in advance, - -Myron Scott - - - -From mscott@sacadia.com Thu Nov 16 17:19:45 2000 -Received: from goldengate.kojoworldwide.com. ([216.133.4.130]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id RAA04315 - for ; Thu, 16 Nov 2000 17:19:43 -0500 (EST) -Received: from localhost (localhost [127.0.0.1]) - by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id OAA11449; - Thu, 16 Nov 2000 14:05:15 -0800 (PST) -Date: Thu, 16 Nov 2000 14:05:15 -0800 (PST) -From: Myron Scott -X-Sender: mscott@goldengate.kojoworldwide.com. -To: Bruce Momjian -cc: "Mikheev, Vadim" , Tom Lane -Subject: Re: Please help with some advice -In-Reply-To: <200011160533.AAA27886@candle.pha.pa.us> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Status: OR - -Bruce Momjian wrote: - ->I am curious how you isolated each thread. It seems we pretty much ->assume all our memory is controlled by a single query in the process. - - -I moved all global variables to a thread global variable which is accessed -by the method GetEnv(). Which looks like this - -Env* GetEnv(void) { - Env* env; - thr_getspecific(*envkey,(void*)&env); - return env; -} - -The Env struct includes the CurrentMemoryContext, TopMemoryContext, -PortalHeapMemory for each instance of a connection (one thread per -connection). So, for example, -EndPortalAllocMode uses GetEnv()->CurrentMemoryContext - -void -EndPortalAllocMode() -{ - PortalHeapMemory context; - - AssertState(PortalManagerEnabled); - AssertState(IsA(GetEnv()->CurrentMemoryContext, -PortalHeapMemory)); - - context = (PortalHeapMemory) GetEnv()->CurrentMemoryContext; - AssertState(PointerIsValid(context->block)); /* XXX -Trap(...) */ - - /* free current mode */ - AllocSetReset(&HEAPMEMBLOCK(context)->setData); - MemoryContextFree((MemoryContext) -PortalHeapMemoryGetVariableMemory(context), - context->block); - - /* restore previous mode */ - context->block = FixedStackPop(&context->stackData); -} - - - - -From vmikheev@SECTORBASE.COM Thu Nov 16 17:23:22 2000 -Received: from sectorbase2.sectorbase.com ([208.48.122.131]) - by candle.pha.pa.us (8.9.0/8.9.0) with SMTP id RAA04562 - for ; Thu, 16 Nov 2000 17:23:21 -0500 (EST) -Received: by sectorbase2.sectorbase.com with Internet Mail Service (5.5.2650.21) - id ; Thu, 16 Nov 2000 14:05:24 -0800 -Message-ID: <8F4C99C66D04D4118F580090272A7A234D318D@sectorbase1.sectorbase.com> -From: "Mikheev, Vadim" -To: "'Myron Scott'" , - Bruce Momjian - -Cc: Tom Lane -Subject: RE: Please help with some advice -Date: Thu, 16 Nov 2000 14:09:30 -0800 -MIME-Version: 1.0 -X-Mailer: Internet Mail Service (5.5.2650.21) -Content-Type: text/plain; - charset="iso-8859-1" -Status: ORr - -I think the question do we want to make backend multy-threaded -should be discussed in hackers. - -Vadim - -> -----Original Message----- -> From: Myron Scott [mailto:mscott@sacadia.com] -> Sent: Thursday, November 16, 2000 2:05 PM -> To: Bruce Momjian -> Cc: Mikheev, Vadim; Tom Lane -> Subject: Re: Please help with some advice -> -> -> Bruce Momjian wrote: -> -> >I am curious how you isolated each thread. It seems we pretty much -> >assume all our memory is controlled by a single query in the process. -> -> -> -> I moved all global variables to a thread global variable -> which is accessed -> by the method GetEnv(). Which looks like this -> -> Env* GetEnv(void) { -> Env* env; -> thr_getspecific(*envkey,(void*)&env); -> return env; -> } -> -> The Env struct includes the CurrentMemoryContext, TopMemoryContext, -> PortalHeapMemory for each instance of a connection (one thread per -> connection). So, for example, -> EndPortalAllocMode uses GetEnv()->CurrentMemoryContext -> -> void -> EndPortalAllocMode() -> { -> PortalHeapMemory context; -> -> AssertState(PortalManagerEnabled); -> AssertState(IsA(GetEnv()->CurrentMemoryContext, -> PortalHeapMemory)); -> -> context = (PortalHeapMemory) GetEnv()->CurrentMemoryContext; -> AssertState(PointerIsValid(context->block)); /* XXX -> Trap(...) */ -> -> /* free current mode */ -> AllocSetReset(&HEAPMEMBLOCK(context)->setData); -> MemoryContextFree((MemoryContext) -> PortalHeapMemoryGetVariableMemory(context), -> context->block); -> -> /* restore previous mode */ -> context->block = FixedStackPop(&context->stackData); -> } -> -> -> - -From mscott@sacadia.com Thu Nov 16 22:16:38 2000 -Received: from goldengate.kojoworldwide.com. ([216.133.4.130]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id WAA14638 - for ; Thu, 16 Nov 2000 22:16:36 -0500 (EST) -Received: from localhost (localhost [127.0.0.1]) - by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id TAA11874; - Thu, 16 Nov 2000 19:04:48 -0800 (PST) -Date: Thu, 16 Nov 2000 19:04:48 -0800 (PST) -From: Myron Scott -X-Sender: mscott@goldengate.kojoworldwide.com. -To: Bruce Momjian -cc: "Mikheev, Vadim" , Tom Lane -Subject: Re: Please help with some advice -In-Reply-To: <200011170156.UAA11438@candle.pha.pa.us> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Status: ORr - -Thanks very much, I will post to hackers. - -Myron - - - -From pgsql-hackers-owner+M2691@postgresql.org Tue Jan 2 00:30:20 2001 -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id AAA08195 - for ; Tue, 2 Jan 2001 00:30:19 -0500 (EST) -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f025UjL33335; - Tue, 2 Jan 2001 00:30:45 -0500 (EST) - (envelope-from pgsql-hackers-owner+M2691@postgresql.org) -Received: from mailsys01.intnet.net (tmail.wwc.com [198.252.32.143] (may be forged)) - by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f025UTL33232 - for ; Tue, 2 Jan 2001 00:30:32 -0500 (EST) - (envelope-from mscott@sacadia.com) -Received: from [206.112.108.0] (HELO sacadia.com) - by mailsys01.intnet.net (CommuniGate Pro SMTP 3.3.2) - with ESMTP id 2214231; Tue, 02 Jan 2001 00:29:47 -0500 -Message-ID: <3A5167DB.3050807@sacadia.com> -Date: Mon, 01 Jan 2001 21:32:11 -0800 -From: Myron Scott -Reply-To: mscott@sacadia.com -User-Agent: Mozilla/5.0 (Windows; U; Win98; en-US; m18) Gecko/20001108 Netscape6/6.0 -X-Accept-Language: en -MIME-Version: 1.0 -To: "Ross J. Reedstrom" -CC: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Using Threads? -References: <004401c058fd$fd498d40$f2356880@tracy> <20001204113307.B5871@rice.edu> -Content-Type: text/plain; charset=us-ascii; format=flowed -Content-Transfer-Encoding: 7bit -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -For anyone interested, - -I have posted my multi-threaded version of PostgreSQL here. - -http://www.sacadia.com/mtpg.html - -It is based on 7.0.2 and the TAO CORBA ORB which is here. - -http://www.cs.wustl.edu/~schmidt/TAO.html - -Myron Scott -mkscott@sacadia.com - - - -From bright@fw.wintelcom.net Tue Jan 2 03:02:28 2001 -Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id DAA16169 - for ; Tue, 2 Jan 2001 03:02:27 -0500 (EST) -Received: (from bright@localhost) - by fw.wintelcom.net (8.10.0/8.10.0) id f0282Vm10623; - Tue, 2 Jan 2001 00:02:31 -0800 (PST) -Date: Tue, 2 Jan 2001 00:02:31 -0800 -From: Alfred Perlstein -To: Bruce Momjian -Cc: Tom Lane , pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Assuming that TAS() will succeed the first time is verboten -Message-ID: <20010102000230.C19572@fw.wintelcom.net> -References: <9850.978067943@sss.pgh.pa.us> <200101020759.CAA15836@candle.pha.pa.us> -Mime-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -User-Agent: Mutt/1.2.5i -In-Reply-To: <200101020759.CAA15836@candle.pha.pa.us>; from pgman@candle.pha.pa.us on Tue, Jan 02, 2001 at 02:59:20AM -0500 -Status: OR - -* Bruce Momjian [010101 23:59] wrote: -> > Alfred Perlstein writes: -> > > One trick that may help is calling sched_yield(2) on a lock miss, -> > > it's a POSIX call and quite new so you'd need a 'configure' test -> > > for it. -> > -> > The author of the current s_lock code seems to have thought that -> > select() with a zero delay would do the equivalent of sched_yield(). -> > I'm not sure if that's true on very many kernels, if indeed any... -> > -> > I doubt we could buy much by depending on sched_yield(); if you want -> > to assume POSIX facilities, ISTM you might as well go for user-space -> > semaphores and forget the whole TAS mechanism. -> -> -> Another issue is that sched_yield brings in the pthreads library/hooks -> on some OS's, which we certainly want to avoid. - -I know it's a major undertaking, but since the work is sort of done, -have you guys considered the port to solaris threads and seeing about -making a pthreads port of that? - -I know it would probably get you considerable gains under Windows -at the expense of dropping some really really legacy system. - -Or you could do what apache (is rumored) does and have it do either -threads or processes or both... - --- --Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] -"I have the heart of a child; I keep it in a jar on my desk." - -From pgsql-hackers-owner+M4275@postgresql.org Mon Feb 5 21:45:00 2001 -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id VAA09262 - for ; Mon, 5 Feb 2001 21:44:59 -0500 (EST) -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f162ixx00920; - Mon, 5 Feb 2001 21:44:59 -0500 (EST) - (envelope-from pgsql-hackers-owner+M4275@postgresql.org) -Received: from goldengate.kojoworldwide.com. ([216.133.4.130]) - by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f162fSx00595 - for ; Mon, 5 Feb 2001 21:41:29 -0500 (EST) - (envelope-from mscott@sacadia.com) -Received: from localhost (localhost [127.0.0.1]) - by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id SAA03298 - for ; Mon, 5 Feb 2001 18:25:05 -0800 (PST) -Date: Mon, 5 Feb 2001 18:25:05 -0800 (PST) -From: Myron Scott -X-Sender: mscott@goldengate.kojoworldwide.com. -To: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Using Threads? -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -I have put a new version of my multi-threaded -postgresql experiment at - -http://www.sacadia.com/mtpg.html - -This one actually works. I have added a server -based on omniORB, a CORBA 2.3 ORB from ATT. It - is much smaller than TAO and uses the thread per -connection model. I haven't added the java side -of the JNI interface yet but the C++ side is there. - -It's still not stable but it is much better than -the last. - -Myron Scott -mkscott@sacadia.com - - - - - -From pgsql-hackers-owner+M4304@postgresql.org Tue Feb 6 10:24:21 2001 -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id KAA22027 - for ; Tue, 6 Feb 2001 10:24:20 -0500 (EST) -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f16FOBx97182; - Tue, 6 Feb 2001 10:24:11 -0500 (EST) - (envelope-from pgsql-hackers-owner+M4304@postgresql.org) -Received: from goldengate.kojoworldwide.com. ([216.133.4.130]) - by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f16FLWx96814 - for ; Tue, 6 Feb 2001 10:21:33 -0500 (EST) - (envelope-from mscott@sacadia.com) -Received: from localhost (localhost [127.0.0.1]) - by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id HAA04170; - Tue, 6 Feb 2001 07:05:04 -0800 (PST) -Date: Tue, 6 Feb 2001 07:05:04 -0800 (PST) -From: Myron Scott -X-Sender: mscott@goldengate.kojoworldwide.com. -To: Karel Zak -cc: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Using Threads -In-Reply-To: -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - -> -> Sorry I haven't time to see and test your experiment, -> but I have a question. How you solve memory management? -> The current mmgr is based on global variable -> CurrentMemoryContext that is very often changed and used. -> Use you for this locks? If yes it is probably problematic -> point for perfomance. -> -> Karel -> - -There are many many globals I had to work around including all the memory -management stuff. I basically threw everything into and "environment" -variable which I stored in a thread specific using thr_setspecific. - -Performance is acually very good for what I am doing. I was able to batch -commit transactions which cuts down on fsync calls, use prepared -statements from my client using CORBA, and the various locking calls for -the threads (cond_wait,mutex_lock, and sema_wait) seem pretty fast. I did -some performance tests for inserts - -20 clients, 900 inserts per client, 1 insert per transaction, 4 different -tables. - -7.0.2 About 10:52 average completion -multi-threaded 2:42 average completion -7.1beta3 1:13 average completion - -If I increased the number of inserts per transaction, multi-threaded got -closer to 7.1 for inserts. I haven't tested other other types of -commands -yet. - - -Myron Scott -mkscott@sacadia.com - - -From pgsql-hackers-owner+M4313@postgresql.org Tue Feb 6 12:32:00 2001 -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id MAA29163 - for ; Tue, 6 Feb 2001 12:31:59 -0500 (EST) -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f16HVox17454; - Tue, 6 Feb 2001 12:31:51 -0500 (EST) - (envelope-from pgsql-hackers-owner+M4313@postgresql.org) -Received: from ara.zf.jcu.cz (ara.zf.jcu.cz [160.217.161.4]) - by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f16HV6x17323 - for ; Tue, 6 Feb 2001 12:31:06 -0500 (EST) - (envelope-from zakkr@zf.jcu.cz) -Received: from localhost (zakkr@localhost) - by ara.zf.jcu.cz (8.9.3/8.9.3/Debian 8.9.3-21) with SMTP id SAA03980; - Tue, 6 Feb 2001 18:31:02 +0100 -Date: Tue, 6 Feb 2001 18:31:02 +0100 (CET) -From: Karel Zak -To: Myron Scott -cc: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Using Threads -In-Reply-To: -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - -On Tue, 6 Feb 2001, Myron Scott wrote: - -> There are many many globals I had to work around including all the memory -> management stuff. I basically threw everything into and "environment" -> variable which I stored in a thread specific using thr_setspecific. - - Yes, it's good. I working on multi-thread application server -(http://mape.jcu.cz) and I use for this project some things from PG (like -mmgr), I planning use same solution. - -> Performance is acually very good for what I am doing. I was able to batch -> commit transactions which cuts down on fsync calls, use prepared -> statements from my client using CORBA, and the various locking calls for -> the threads (cond_wait,mutex_lock, and sema_wait) seem pretty fast. I did -> some performance tests for inserts -> -> 20 clients, 900 inserts per client, 1 insert per transaction, 4 different -> tables. -> -> 7.0.2 About 10:52 average completion -> multi-threaded 2:42 average completion -> 7.1beta3 1:13 average completion - -It is very very good for time for 7.1, already look forward to 7.2! :-) - - BTW, I not sure if you anytime in future will see threads in -official PostgreSQL and if you spending time on relevant things (IMHO). - - Karel - - - - - - -From pgsql-hackers-owner+M4304@postgresql.org Tue Feb 6 10:24:21 2001 -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id KAA22027 - for ; Tue, 6 Feb 2001 10:24:20 -0500 (EST) -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f16FOBx97182; - Tue, 6 Feb 2001 10:24:11 -0500 (EST) - (envelope-from pgsql-hackers-owner+M4304@postgresql.org) -Received: from goldengate.kojoworldwide.com. ([216.133.4.130]) - by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f16FLWx96814 - for ; Tue, 6 Feb 2001 10:21:33 -0500 (EST) - (envelope-from mscott@sacadia.com) -Received: from localhost (localhost [127.0.0.1]) - by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id HAA04170; - Tue, 6 Feb 2001 07:05:04 -0800 (PST) -Date: Tue, 6 Feb 2001 07:05:04 -0800 (PST) -From: Myron Scott -X-Sender: mscott@goldengate.kojoworldwide.com. -To: Karel Zak -cc: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Using Threads -In-Reply-To: -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - -> -> Sorry I haven't time to see and test your experiment, -> but I have a question. How you solve memory management? -> The current mmgr is based on global variable -> CurrentMemoryContext that is very often changed and used. -> Use you for this locks? If yes it is probably problematic -> point for perfomance. -> -> Karel -> - -There are many many globals I had to work around including all the memory -management stuff. I basically threw everything into and "environment" -variable which I stored in a thread specific using thr_setspecific. - -Performance is acually very good for what I am doing. I was able to batch -commit transactions which cuts down on fsync calls, use prepared -statements from my client using CORBA, and the various locking calls for -the threads (cond_wait,mutex_lock, and sema_wait) seem pretty fast. I did -some performance tests for inserts - -20 clients, 900 inserts per client, 1 insert per transaction, 4 different -tables. - -7.0.2 About 10:52 average completion -multi-threaded 2:42 average completion -7.1beta3 1:13 average completion - -If I increased the number of inserts per transaction, multi-threaded got -closer to 7.1 for inserts. I haven't tested other other types of -commands -yet. - - -Myron Scott -mkscott@sacadia.com - - -From lamar.owen@wgcr.org Thu Jun 28 11:14:10 2001 -Return-path: -Received: from www.wgcr.org (IDENT:root@www.wgcr.org [206.74.232.194]) - by candle.pha.pa.us (8.10.1/8.10.1) with ESMTP id f5SFE9U18758 - for ; Thu, 28 Jun 2001 11:14:09 -0400 (EDT) -Received: from lowen.wgcr.org (IDENT:lowen@[10.1.2.3]) - by www.wgcr.org (8.9.3/8.9.3/WGCR) with SMTP id LAA11879; - Thu, 28 Jun 2001 11:14:14 -0400 -Content-Type: text/plain; - charset="iso-8859-1" -From: Lamar Owen -To: Bruce Momjian -Subject: Process weight (was:Re: [GENERAL] Re: Red Hat to support PostgreSQL) -Date: Thu, 28 Jun 2001 11:14:09 -0400 -X-Mailer: KMail [version 1.2] -References: <200106272258.f5RMwIb26959@candle.pha.pa.us> -In-Reply-To: <200106272258.f5RMwIb26959@candle.pha.pa.us> -MIME-Version: 1.0 -Message-ID: <01062811140902.01118@lowen.wgcr.org> -Content-Transfer-Encoding: 8bit -Status: ORr - -On Wednesday 27 June 2001 18:58, Bruce Momjian wrote: -> > I had almost given up on using Postgres for this system because under -> > Solaris, it just couldn't cut it (MySQL could do the work with one CPU -> > while Postgres took up even more CPU and required *both* CPUs to be -> > enabled), but when we moved the system to a Linux box, things worked -> > much better. - -> Ah, back to a PostgreSQL topic. :-) - -> My guess on this one is that Solaris is slower for PostgreSQL because -> process switching is _much_ heavier on Solaris than other OS's. This is -> because of the way they implemented processes in SVr4. They got quite -> heavy, almost requiring kernel threads so you weren't switching -> processes all the time. - -Now, the question of the week: -Is supporting a thread model for an inefficient OS a desirable thing to do, -when more efficient OS kernels are available such as FreeBSD 4.x and Linux -2.4? My opinion is that our existing model, when used with a -connection-pooling frontend, is rather efficient. (Yes, I use a -connection-pooling frontend. Performance is rather nice, and I don't have to -have a full backend spawned for every page hit.) - -In fact, on a Linux box threads show as processes. While I know that the -kernel actually supports themin a slightly different manner than processes, -they have more similarities than differences. - -However, even on OS's where threads are supported, the mechanism to support -those threads must be an efficient one -- not all pthreads libraries are -created equal. Many are frontends (expensive ones, at that) for plain old -processes. - -Does anyone know of a resource that details the 'weight' of processes for our -supported platforms? [reply off-list -- I'll be glad to summarize responses -to HACKERS, ADMIN, or PORTS, as appropriate, if desired.] --- -Lamar Owen -WGCR Internet Radio -1 Peter 4:11 - -From pgsql-hackers-owner+M13599=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 17:25:32 2001 -Return-path: -Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8QLPWo07589 - for ; Wed, 26 Sep 2001 17:25:32 -0400 (EDT) -Received: from postgresql.org (webmail.postgresql.org [216.126.85.28]) - by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8QLPf405606 - for ; Wed, 26 Sep 2001 16:25:41 -0500 (CDT) - (envelope-from pgsql-hackers-owner+M13599=candle.pha.pa.us=pgman@postgresql.org) -Received: from gromit.dotclick.com (ipn9-f8366.net-resource.net [216.204.83.66]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id f8QKj3h82020 - for ; Wed, 26 Sep 2001 16:45:03 -0400 (EDT) - (envelope-from markw@mohawksoft.com) -Received: from mohawksoft.com (IDENT:markw@localhost.localdomain [127.0.0.1]) - by gromit.dotclick.com (8.9.3/8.9.3) with ESMTP id QAA23693; - Wed, 26 Sep 2001 16:43:02 -0400 -Message-ID: <3BB23DD6.E86AF327@mohawksoft.com> -Date: Wed, 26 Sep 2001 16:43:02 -0400 -From: mlw -X-Mailer: Mozilla 4.75 [en] (X11; U; Linux 2.4.2 i686) -X-Accept-Language: en -MIME-Version: 1.0 -To: "D. Hageman" , - "pgsql-hackers@postgresql.org" -Subject: Re: [HACKERS] Spinlock performance improvement proposal -References: -Content-Type: text/plain; charset=us-ascii -Content-Transfer-Encoding: 7bit -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -"D. Hageman" wrote: - -> The plan for the new spinlocks does look like it has some potential. My -> only comment in regards to permformance when we start looking at SMP -> machines is ... it is my belief that getting a true threaded backend may -> be the only way to get the full potential out of SMP machines. I see that -> is one of the things to experiment with on the TODO list and I have seen -> some people have messed around already with this using Solaris threads. -> It should probably be attempted with pthreads if PostgreSQL is going to -> keep some resemblance of cross-platform compatibility. At that time, it -> would probably be easier to go in and clean up some stuff for the -> implementation of other TODO items (put in the base framework for more -> complex future items) as threading the backend would take a little bit of -> ideology shift. - -I can only think of two objectives for threading. (1) running the various -connections in their own thread instead of their own process. (2) running -complex queries across multiple threads. - -For item (1) I see no value to this. It is a lot of work with no tangible -benefit. If you have an old fashion pthreads implementation, it will hurt -performance because are scheduled within the single process's time slice.. If -you have a newer kernel scheduled implementation, then you will have the same -scheduling as separate processes. The only thing you will need to do is -switch your brain from figuring out how to share data, to trying to figure -out how to isolate data. A multithreaded implementation lacks many of the -benefits and robustness of a multiprocess implementation. - -For item (2) I can see how that could speed up queries in a low utilization -system, and that would be cool, but in a server that is under load, threading -the queries probably be less efficient. - - ----------------------------(end of broadcast)--------------------------- -TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org - -From pgsql-hackers-owner+M13604=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 18:40:26 2001 -Return-path: -Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8QMePo13437 - for ; Wed, 26 Sep 2001 18:40:25 -0400 (EDT) -Received: from postgresql.org (webmail.postgresql.org [216.126.85.28]) - by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8QMeZ417944 - for ; Wed, 26 Sep 2001 17:40:35 -0500 (CDT) - (envelope-from pgsql-hackers-owner+M13604=candle.pha.pa.us=pgman@postgresql.org) -Received: from foghorn.airs.com (foghorn.airs.com [63.201.54.26]) - by postgresql.org (8.11.3/8.11.4) with SMTP id f8QM59h01247 - for ; Wed, 26 Sep 2001 18:05:09 -0400 (EDT) - (envelope-from ian@airs.com) -Received: (qmail 10089 invoked by uid 10); 26 Sep 2001 22:04:49 -0000 -Received: (qmail 6837 invoked by uid 269); 26 Sep 2001 22:04:41 -0000 -Mail-Followup-To: markw@mohawksoft.com, - pgsql-hackers@postgresql.org, - dhageman@dracken.com -To: "D. Hageman" -cc: mlw , - "pgsql-hackers@postgresql.org" -Subject: Re: [HACKERS] Spinlock performance improvement proposal -References: -From: Ian Lance Taylor -Date: 26 Sep 2001 15:04:41 -0700 -In-Reply-To: -Message-ID: -Lines: 45 -User-Agent: Gnus/5.0808 (Gnus v5.8.8) Emacs/20.7 -MIME-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -"D. Hageman" writes: - -> > you have a newer kernel scheduled implementation, then you will have the same -> > scheduling as separate processes. The only thing you will need to do is -> > switch your brain from figuring out how to share data, to trying to figure -> > out how to isolate data. A multithreaded implementation lacks many of the -> > benefits and robustness of a multiprocess implementation. -> -> Save for the fact that the kernel can switch between threads faster then -> it can switch processes considering threads share the same address space, -> stack, code, etc. If need be sharing the data between threads is much -> easier then sharing between processes. - -When using a kernel threading model, it's not obvious to me that the -kernel will switch between threads much faster than it will switch -between processes. As far as I can see, the only potential savings is -not reloading the pointers to the page tables. That is not nothing, -but it is also not a lot. - -> I can't comment on the "isolate data" line. I am still trying to figure -> that one out. - -Sometimes you need data which is specific to a particular thread. -Basically, you have to look at every global variable in the Postgres -backend, and determine whether to share it among all threads or to -make it thread-specific. In other words, you have to take extra steps -to isolate the data within the thread. This is the reverse of the -current situation, in which you have to take extra steps to share data -among all backend processes. - -> That last line is a troll if I every saw it ;-) I will agree that threads -> isn't for everything and that it has costs just like everything else. Let -> me stress that last part - like everything else. Certain costs exist in -> the present model, nothing is - how should we say ... perfect. - -When writing in C, threading inevitably loses robustness. Erratic -behaviour by one thread, perhaps in a user defined function, can -subtly corrupt the entire system, rather than just that thread. Part -of defensive programming is building barriers between different parts -of a system. Process boundaries are a powerful barrier. - -(Actually, though, Postgres is already vulnerable to erratic behaviour -because any backend process can corrupt the shared buffer pool.) - -Ian - ----------------------------(end of broadcast)--------------------------- -TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org - -From pgsql-hackers-owner+M13605=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 18:54:58 2001 -Return-path: -Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8QMsvo14061 - for ; Wed, 26 Sep 2001 18:54:57 -0400 (EDT) -Received: from postgresql.org (webmail.postgresql.org [216.126.85.28]) - by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8QMt7420740 - for ; Wed, 26 Sep 2001 17:55:07 -0500 (CDT) - (envelope-from pgsql-hackers-owner+M13605=candle.pha.pa.us=pgman@postgresql.org) -Received: from goldengate.kojoworldwide.com. ([216.133.4.130]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id f8QMOPh04333 - for ; Wed, 26 Sep 2001 18:24:26 -0400 (EDT) - (envelope-from mscott@sacadia.com) -Received: from localhost (localhost [127.0.0.1]) - by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id PAA00633 - for ; Wed, 26 Sep 2001 15:03:00 -0700 (PDT) -Date: Wed, 26 Sep 2001 15:03:00 -0700 (PDT) -From: Myron Scott -X-Sender: mscott@goldengate.kojoworldwide.com. -To: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Spinlock performance improvement proposal -In-Reply-To: <3BB23DD6.E86AF327@mohawksoft.com> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - - -On Wed, 26 Sep 2001, mlw wrote: - -> I can only think of two objectives for threading. (1) running the various -> connections in their own thread instead of their own process. (2) running -> complex queries across multiple threads. -> - -I did a multi-threaded version of 7.0.2 using Solaris threads about a year -ago in order to try -and get multiple backend connections working under one java process using -jni. I used the thread per connection model. - -I eventually got it working, but it was/is very messy ( there were global -variables everywhere! ). Anyway, I was able to get a pretty good speed up -on inserts by scheduling buffer writes from multiple connections on one -common writing thread. - -I also got some other features that were important to me at the time. - -1. True prepared statements under java with bound input and output -variables -2. Better system utilization - a. fewer Solaris lightweight processes mapped to threads. - b. Fewer open files per postgres installation -3. Automatic vacuums when system activity is low by a daemon thread. - -but there were some drawbacks... One rogue thread or bad user -function could take down all connections for that process. This -was and seems to still be the major drawback to using threads. - - -Myron Scott -mscott@sacadia.com - - ----------------------------(end of broadcast)--------------------------- -TIP 5: Have you checked our extensive FAQ? - -http://www.postgresql.org/users-lounge/docs/faq.html - -From pgsql-hackers-owner+M13602=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 17:45:26 2001 -Return-path: -Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8QLjQo08483 - for ; Wed, 26 Sep 2001 17:45:26 -0400 (EDT) -Received: from postgresql.org (webmail.postgresql.org [216.126.85.28]) - by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8QLjY409914 - for ; Wed, 26 Sep 2001 16:45:35 -0500 (CDT) - (envelope-from pgsql-hackers-owner+M13602=candle.pha.pa.us=pgman@postgresql.org) -Received: from typhon.dracken.com (dv07m61.lawrence.ks.us [24.124.61.35]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id f8QLGDh91021 - for ; Wed, 26 Sep 2001 17:16:13 -0400 (EDT) - (envelope-from dhageman@dracken.com) -Received: from localhost (dhageman@localhost) - by typhon.dracken.com (8.11.4/8.11.4) with ESMTP id f8QLEMY01973; - Wed, 26 Sep 2001 16:14:22 -0500 -X-Authentication-Warning: typhon.dracken.com: dhageman owned process doing -bs -Date: Wed, 26 Sep 2001 16:14:22 -0500 (CDT) -From: "D. Hageman" -To: mlw -cc: "pgsql-hackers@postgresql.org" -Subject: Re: [HACKERS] Spinlock performance improvement proposal -In-Reply-To: <3BB23DD6.E86AF327@mohawksoft.com> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: ORr - -On Wed, 26 Sep 2001, mlw wrote: -> -> I can only think of two objectives for threading. (1) running the various -> connections in their own thread instead of their own process. (2) running -> complex queries across multiple threads. -> -> For item (1) I see no value to this. It is a lot of work with no tangible -> benefit. If you have an old fashion pthreads implementation, it will hurt -> performance because are scheduled within the single process's time slice.. - -Old fashion ... as in a userland library that implements POSIX threads? -Well, I would agree. However, most *modern* implementations are done in -the kernel or kernel and userland coop model and don't have this -limitation (as you mention later in your e-mail). You have kinda hit on -one of my gripes about computers in general. At what point in time does -one say something is obsolete or too old to support anymore - that it -hinders progress instead of adding a "feature"? - -> you have a newer kernel scheduled implementation, then you will have the same -> scheduling as separate processes. The only thing you will need to do is -> switch your brain from figuring out how to share data, to trying to figure -> out how to isolate data. A multithreaded implementation lacks many of the -> benefits and robustness of a multiprocess implementation. - -Save for the fact that the kernel can switch between threads faster then -it can switch processes considering threads share the same address space, -stack, code, etc. If need be sharing the data between threads is much -easier then sharing between processes. - -I can't comment on the "isolate data" line. I am still trying to figure -that one out. - -That last line is a troll if I every saw it ;-) I will agree that threads -isn't for everything and that it has costs just like everything else. Let -me stress that last part - like everything else. Certain costs exist in -the present model, nothing is - how should we say ... perfect. - -> For item (2) I can see how that could speed up queries in a low utilization -> system, and that would be cool, but in a server that is under load, threading -> the queries probably be less efficient. - -Well, I don't follow your logic and you didn't give any substance to back -up your claim. I am willing to listen. - -Another thought ... Oracle uses threads doesn't it or at least it has a -single processor and multi-processor version last time I knew ... which do -they claim is better? (Not saying that Oracle's proclimation of what is -good and what is not matters, but it is good for another view point). - --- -//========================================================\\ -|| D. Hageman || -\\========================================================// - - ----------------------------(end of broadcast)--------------------------- -TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org - -From pgsql-hackers-owner+M13607=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 19:14:59 2001 -Return-path: -Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8QNExo15536 - for ; Wed, 26 Sep 2001 19:14:59 -0400 (EDT) -Received: from postgresql.org (webmail.postgresql.org [216.126.85.28]) - by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8QNF8423944 - for ; Wed, 26 Sep 2001 18:15:09 -0500 (CDT) - (envelope-from pgsql-hackers-owner+M13607=candle.pha.pa.us=pgman@postgresql.org) -Received: from belphigor.mcnaught.org ([216.151.155.121]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id f8QMe3h07256 - for ; Wed, 26 Sep 2001 18:40:04 -0400 (EDT) - (envelope-from doug@wireboard.com) -Received: (from doug@localhost) - by belphigor.mcnaught.org (8.11.6/8.9.3) id f8QMdkB05502; - Wed, 26 Sep 2001 18:39:46 -0400 -X-Authentication-Warning: belphigor.mcnaught.org: doug set sender to doug@wireboard.com using -f -To: "D. Hageman" -cc: mlw , - "pgsql-hackers@postgresql.org" -Subject: Re: [HACKERS] Spinlock performance improvement proposal -References: -From: Doug McNaught -Date: 26 Sep 2001 18:39:44 -0400 -In-Reply-To: "D. Hageman"'s message of "Wed, 26 Sep 2001 16:14:22 -0500 (CDT)" -Message-ID: -Lines: 26 -User-Agent: Gnus/5.0806 (Gnus v5.8.6) XEmacs/21.1 (20 Minutes to Nikko) -MIME-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -"D. Hageman" writes: - -> Save for the fact that the kernel can switch between threads faster then -> it can switch processes considering threads share the same address space, -> stack, code, etc. If need be sharing the data between threads is much -> easier then sharing between processes. - -This depends on your system. Solaris has a huge difference between -thread and process context switch times, whereas Linux has very little -difference (and in fact a Linux process context switch is about as -fast as a Solaris thread switch on the same hardware--Solaris is just -a pig when it comes to process context switching). - -> I can't comment on the "isolate data" line. I am still trying to figure -> that one out. - -I think his point is one of clarity and maintainability. When a -task's data is explicitly shared (via shared memory of some sort) it's -fairly clear when you're accessing shared data and need to worry about -locking. Whereas when all data is shared by default (as with threads) -it's very easy to miss places where threads can step on each other. - --Doug --- -In a world of steel-eyed death, and men who are fighting to be warm, -Come in, she said, I'll give you shelter from the storm. -Dylan - ----------------------------(end of broadcast)--------------------------- -TIP 2: you can get off all lists at once with the unregister command - (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) - -From pgsql-hackers-owner+M13611=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 21:05:02 2001 -Return-path: -Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8R152o22010 - for ; Wed, 26 Sep 2001 21:05:02 -0400 (EDT) -Received: from postgresql.org (webmail.postgresql.org [216.126.85.28]) - by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8R158430261 - for ; Wed, 26 Sep 2001 20:05:08 -0500 (CDT) - (envelope-from pgsql-hackers-owner+M13611=candle.pha.pa.us=pgman@postgresql.org) -Received: from sss.pgh.pa.us ([192.204.191.242]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id f8R0lgh29430 - for ; Wed, 26 Sep 2001 20:47:42 -0400 (EDT) - (envelope-from tgl@sss.pgh.pa.us) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id f8R0kpK14707; - Wed, 26 Sep 2001 20:46:51 -0400 (EDT) -To: Ian Lance Taylor -cc: "D. Hageman" , mlw , - "pgsql-hackers@postgresql.org" -Subject: Re: [HACKERS] Spinlock performance improvement proposal -In-Reply-To: -References: -Comments: In-reply-to Ian Lance Taylor - message dated "26 Sep 2001 15:04:41 -0700" -Date: Wed, 26 Sep 2001 20:46:51 -0400 -Message-ID: <14704.1001551611@sss.pgh.pa.us> -From: Tom Lane -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -Ian Lance Taylor writes: -> (Actually, though, Postgres is already vulnerable to erratic behaviour -> because any backend process can corrupt the shared buffer pool.) - -Not to mention the other parts of shared memory. - -Nonetheless, our experience has been that cross-backend failures due to -memory clobbers in shared memory are very infrequent --- certainly far -less often than we see localized-to-a-backend crashes. Probably this is -because the shared memory is (a) small compared to the rest of the -address space and (b) only accessed by certain specific modules within -Postgres. - -I'm convinced that switching to a thread model would result in a -significant degradation in our ability to recover from coredump-type -failures, even given the (implausible) assumption that we introduce no -new bugs during the conversion. I'm also *un*convinced that such a -conversion will yield significant performance benefits, unless we -introduce additional cross-thread dependencies (and more fragility -and lock contention) by tactics such as sharing catalog caches across -threads. - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate -subscribe-nomail command to majordomo@postgresql.org so that your -message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M13616=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 23:10:52 2001 -Return-path: -Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8R3Aqo03180 - for ; Wed, 26 Sep 2001 23:10:52 -0400 (EDT) -Received: from postgresql.org (webmail.postgresql.org [216.126.85.28]) - by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8R3B3438816 - for ; Wed, 26 Sep 2001 22:11:03 -0500 (CDT) - (envelope-from pgsql-hackers-owner+M13616=candle.pha.pa.us=pgman@postgresql.org) -Received: from spider.pilosoft.com (p55-222.acedsl.com [160.79.55.222]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id f8R2vCh48923 - for ; Wed, 26 Sep 2001 22:57:12 -0400 (EDT) - (envelope-from alex@pilosoft.com) -Received: from localhost (alexmail@localhost) - by spider.pilosoft.com (8.9.3/8.9.3) with ESMTP id WAA27630; - Wed, 26 Sep 2001 22:58:41 -0400 (EDT) -Date: Wed, 26 Sep 2001 22:58:41 -0400 (EDT) -From: Alex Pilosov -To: "D. Hageman" -cc: "pgsql-hackers@postgresql.org" -Subject: Re: [HACKERS] Spinlock performance improvement proposal -In-Reply-To: -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -On Wed, 26 Sep 2001, D. Hageman wrote: - -> > > Save for the fact that the kernel can switch between threads faster then -> > > it can switch processes considering threads share the same address space, -> > > stack, code, etc. If need be sharing the data between threads is much -> > > easier then sharing between processes. -> > -> > When using a kernel threading model, it's not obvious to me that the -> > kernel will switch between threads much faster than it will switch -> > between processes. As far as I can see, the only potential savings is -> > not reloading the pointers to the page tables. That is not nothing, -> > but it is also - -> > > I can't comment on the "isolate data" line. I am still trying to figure -> > > that one out. -> > -> > Sometimes you need data which is specific to a particular thread. -> -> When you need data that is specific to a thread you use a TSD (Thread -> Specific Data). -Which Linux does not support with a vengeance, to my knowledge. - -As a matter of fact, quote from Linus on the matter was something like -"Solution to slow process switching is fast process switching, not another -kernel abstraction [referring to threads and TSD]". TSDs make -implementation of thread switching complex, and fork() complex. - -The question about threads boils down to: Is there far more data that is -shared than unshared? If yes, threads are better, if not, you'll be -abusing TSD and slowing things down. - -I believe right now, postgresql' model of sharing only things that need to -be shared is pretty damn good. The only slight problem is overhead of -forking another backend, but its still _fast_. - -IMHO, threads would not bring large improvement to postgresql. - - Actually, if I remember, there was someone who ported postgresql (I think -it was 6.5) to be multithreaded with major pain, because the requirement -was to integrate with CORBA. I believe that person posted some benchmarks -which were essentially identical to non-threaded postgres... - --alex - - ----------------------------(end of broadcast)--------------------------- -TIP 6: Have you searched our list archives? - -http://archives.postgresql.org - -From pgsql-hackers-owner+M13619=candle.pha.pa.us=pgman@postgresql.org Thu Sep 27 00:32:55 2001 -Return-path: -Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8R4Wto07075 - for ; Thu, 27 Sep 2001 00:32:55 -0400 (EDT) -Received: from postgresql.org (webmail.postgresql.org [216.126.85.28]) - by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8R4X7444942 - for ; Wed, 26 Sep 2001 23:33:07 -0500 (CDT) - (envelope-from pgsql-hackers-owner+M13619=candle.pha.pa.us=pgman@postgresql.org) -Received: from sss.pgh.pa.us ([192.204.191.242]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id f8R4Jsh61257 - for ; Thu, 27 Sep 2001 00:19:54 -0400 (EDT) - (envelope-from tgl@sss.pgh.pa.us) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id f8R4JLK15406; - Thu, 27 Sep 2001 00:19:21 -0400 (EDT) -To: "D. Hageman" -cc: Alex Pilosov , - "pgsql-hackers@postgresql.org" -Subject: Re: [HACKERS] Spinlock performance improvement proposal -In-Reply-To: -References: -Comments: In-reply-to "D. Hageman" - message dated "Wed, 26 Sep 2001 22:41:39 -0500" -Date: Thu, 27 Sep 2001 00:19:20 -0400 -Message-ID: <15403.1001564360@sss.pgh.pa.us> -From: Tom Lane -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -"D. Hageman" writes: -> If you look at Myron Scott's post today you will see that it had other -> advantages going for it (like auto-vacuum!) and disadvantages ... rogue -> thread corruption (already debated today). - -But note that Myron did a number of things that are (IMHO) orthogonal -to process-to-thread conversion, such as adding prepared statements, -a separate thread/process/whateveryoucallit for buffer writing, ditto -for vacuuming, etc. I think his results cannot be taken as indicative -of the benefits of threads per se --- these other things could be -implemented in a pure process model too, and we have no data with which -to estimate which change bought how much. - -Threading certainly should reduce the context switch time, but this -comes at the price of increased overhead within each context (since -access to thread-local variables is not free). It's by no means -obvious that there's a net win there. - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 6: Have you searched our list archives? - -http://archives.postgresql.org - -From pgsql-hackers-owner+M13621=candle.pha.pa.us=pgman@postgresql.org Thu Sep 27 01:59:44 2001 -Return-path: -Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8R5xio11898 - for ; Thu, 27 Sep 2001 01:59:44 -0400 (EDT) -Received: from postgresql.org (webmail.postgresql.org [216.126.85.28]) - by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8R5xi449748 - for ; Thu, 27 Sep 2001 00:59:45 -0500 (CDT) - (envelope-from pgsql-hackers-owner+M13621=candle.pha.pa.us=pgman@postgresql.org) -Received: from goldengate.kojoworldwide.com. ([216.133.4.130]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id f8R5joh75612 - for ; Thu, 27 Sep 2001 01:45:50 -0400 (EDT) - (envelope-from mscott@sacadia.com) -Received: from localhost (localhost [127.0.0.1]) - by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id WAA01144 - for ; Wed, 26 Sep 2001 22:24:29 -0700 (PDT) -Date: Wed, 26 Sep 2001 22:24:29 -0700 (PDT) -From: Myron Scott -X-Sender: mscott@goldengate.kojoworldwide.com. -To: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Spinlock performance improvement proposal -In-Reply-To: <15403.1001564360@sss.pgh.pa.us> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - - -> But note that Myron did a number of things that are (IMHO) orthogonal - -yes, I did :) - -> to process-to-thread conversion, such as adding prepared statements, -> a separate thread/process/whateveryoucallit for buffer writing, ditto -> for vacuuming, etc. I think his results cannot be taken as indicative -> of the benefits of threads per se --- these other things could be -> implemented in a pure process model too, and we have no data with which -> to estimate which change bought how much. -> - -If you are comparing just process vs. thread, I really don't think I -gained much for performance and ended up with some pretty unmanageable -code. - -The one thing that led to most of the gains was scheduling all the writes -to one thread which, as noted by Tom, you could do on the process model. -Besides, Most of the advantage in doing this was taken away with the -addition of WAL in 7.1. - -The other real gain that I saw with threading was limiting the number of -open files but -that led me to alter much of the file manager in order to synchronize -access to the files which probably slowed things a bit. - -To be honest, I don't think I, personally, -would try this again. I went pretty far off -the beaten path with this thing. It works well for what I am doing -( a limited number of SQL statements run many times over ) but there -probably was a better way. I'm thinking now that I should have tried to -add a CORBA interface for connections. I would have been able to -accomplish my original goals without creating a deadend for myself. - - -Thanks all for a great project, - -Myron -mscott@sacadia.com - - ----------------------------(end of broadcast)--------------------------- -TIP 2: you can get off all lists at once with the unregister command - (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) - -From pgsql-hackers-owner+M13632=candle.pha.pa.us=pgman@postgresql.org Thu Sep 27 10:21:22 2001 -Return-path: -Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8RELLo08607 - for ; Thu, 27 Sep 2001 10:21:21 -0400 (EDT) -Received: from postgresql.org (webmail.postgresql.org [216.126.85.28]) - by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8RELP487000 - for ; Thu, 27 Sep 2001 09:21:26 -0500 (CDT) - (envelope-from pgsql-hackers-owner+M13632=candle.pha.pa.us=pgman@postgresql.org) -Received: from gromit.dotclick.com (ipn9-f8366.net-resource.net [216.204.83.66]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id f8RE49h21870 - for ; Thu, 27 Sep 2001 10:04:09 -0400 (EDT) - (envelope-from markw@mohawksoft.com) -Received: from mohawksoft.com (IDENT:markw@localhost.localdomain [127.0.0.1]) - by gromit.dotclick.com (8.9.3/8.9.3) with ESMTP id KAA24417; - Thu, 27 Sep 2001 10:02:06 -0400 -Message-ID: <3BB3315D.EC99FF65@mohawksoft.com> -Date: Thu, 27 Sep 2001 10:02:05 -0400 -From: mlw -X-Mailer: Mozilla 4.75 [en] (X11; U; Linux 2.4.2 i686) -X-Accept-Language: en -MIME-Version: 1.0 -To: "D. Hageman" -cc: Ian Lance Taylor , - "pgsql-hackers@postgresql.org" -Subject: Re: [HACKERS] Spinlock performance improvement proposal -References: -Content-Type: text/plain; charset=us-ascii -Content-Transfer-Encoding: 7bit -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -"D. Hageman" wrote: - -> On 26 Sep 2001, Ian Lance Taylor wrote: -> > -> > > Save for the fact that the kernel can switch between threads faster then -> > > it can switch processes considering threads share the same address space, -> > > stack, code, etc. If need be sharing the data between threads is much -> > > easier then sharing between processes. -> > -> > When using a kernel threading model, it's not obvious to me that the -> > kernel will switch between threads much faster than it will switch -> > between processes. As far as I can see, the only potential savings is -> > not reloading the pointers to the page tables. That is not nothing, -> > but it is also not a lot. -> -> It is my understanding that avoiding a full context switch of the -> processor can be of a significant advantage. This is especially important -> on processor architectures that can be kinda slow at doing it (x86). I -> will admit that most modern kernels have features that assist software -> packages utilizing the forking model (copy on write for instance). It is -> also my impression that these do a good job. I am the kind of guy that -> looks towards the future (as in a year, year and half or so) and say that -> processors will hopefully get faster at context switching and more and -> more kernels will implement these algorithms to speed up the forking -> model. At the same time, I see more and more processors being shoved into -> a single box and it appears that the threads model works better on these -> type of systems. - -"context" switching happens all the time on a multitasking system. On the x86 -processor, a context switch happens when you call into the kernel. You have to go -through a call-gate to get to a lower privilege ring. "context" switching is very -fast. The operating system dictates how heavy or light a process switch is. Under -Linux (and I believe FreeBSD with Linux threads, or version 4.x ) threads and -processes are virtually identical. The only difference is that the virtual memory -pages are not "copy on write." Process vs thread scheduling is also virtually -identical. - -If you look to the future, then you should accept that process switching should -become more efficient as the operating systems improve. - -> -> > > I can't comment on the "isolate data" line. I am still trying to figure -> > > that one out. -> > -> > Sometimes you need data which is specific to a particular thread. -> -> When you need data that is specific to a thread you use a TSD (Thread -> Specific Data). - -Yes, but Postgres has many global variables. The assumption has always been that -it is a stand-alone process with an explicitly shared paradigm, not implicitly. - -> -> > Basically, you have to look at every global variable in the Postgres -> > backend, and determine whether to share it among all threads or to -> > make it thread-specific. -> -> Yes, if one was to implement threads into PostgreSQL I would think that -> some re-writing would be in order of several areas. Like I said before, -> give a person a chance to restructure things so future TODO items wouldn't -> be so hard to implement. Personally, I like to stay away from global -> variables as much as possible. They just get you into trouble. - -In real live software, software which lives from year to year with active -development, things do get messy. There are always global variables involved in a -program. Efforts, of course, should be made to keep them to a minimum, but the -reality is that they always happen. - -Also, the very structure of function calls may need to change when going from a -process model to a threaded model. Functions never before reentrant are now be -reentrant, think about that. That is a huge undertaking. Every single function -may need to be examined for thread safety, with little benefit. - -> -> > > That last line is a troll if I every saw it ;-) I will agree that threads -> > > isn't for everything and that it has costs just like everything else. Let -> > > me stress that last part - like everything else. Certain costs exist in -> > > the present model, nothing is - how should we say ... perfect. -> > -> > When writing in C, threading inevitably loses robustness. Erratic -> > behaviour by one thread, perhaps in a user defined function, can -> > subtly corrupt the entire system, rather than just that thread. Part -> > of defensive programming is building barriers between different parts -> > of a system. Process boundaries are a powerful barrier. -> -> I agree with everything you wrote above except for the first line. My -> only comment is that process boundaries are only *truely* a powerful -> barrier if the processes are different pieces of code and are not -> dependent on each other in crippling ways. Forking the same code with the -> bug in it - and only 1 in 5 die - is still 4 copies of buggy code running -> on your system ;-) - -This is simply not true. All software has bugs, it is an undeniable fact. Some -bugs are more likely to be hit than others. 5 processes , when one process hits a -bug, that does not mean the other 4 will hit the same bug. Obscure bugs kill -software all the time, the trick is to minimize the impact. Software is not -perfect, assuming it can be is a mistake. - - - - - ----------------------------(end of broadcast)--------------------------- -TIP 5: Have you checked our extensive FAQ? - -http://www.postgresql.org/users-lounge/docs/faq.html - -From pgsql-hackers-owner+M16320=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 10:16:20 2001 -Return-path: -Received: from west.navpoint.com (west.navpoint.com [207.106.42.13]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6FGJZ29347 - for ; Thu, 6 Dec 2001 10:16:19 -0500 (EST) -Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) - by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6FGIE25797 - for ; Thu, 6 Dec 2001 10:16:18 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6F8MR55154 - for ; Thu, 6 Dec 2001 09:12:12 -0600 (CST) - (envelope-from pgsql-hackers-owner+M16320=candle.pha.pa.us=pgman@postgresql.org) -Received: from dendrite.sacadia.com (adsl-64-168-22-137.dsl.snfc21.pacbell.net [64.168.22.137]) - by postgresql.org (8.11.3/8.11.4) with ESMTP id fB3NNfm32380 - for ; Mon, 3 Dec 2001 18:23:42 -0500 (EST) - (envelope-from mkscott@sacadia.com) -Received: from sacadia.com (localhost [127.0.0.1]) - by dendrite.sacadia.com (8.10.2+Sun/8.10.2) with ESMTP id fB3NKiK16816 - for ; Mon, 3 Dec 2001 15:20:44 -0800 (PST) -Message-ID: <3C0C08CC.917CC04B@sacadia.com> -Date: Mon, 03 Dec 2001 15:20:44 -0800 -From: mkscott@sacadia.com -X-Mailer: Mozilla 4.78 [en] (X11; U; SunOS 5.8 sun4u) -X-Accept-Language: en -MIME-Version: 1.0 -To: PostgreSQL Hackers Mailing List -Subject: [HACKERS] Using Threads (again) -Content-Type: text/plain; charset=us-ascii -Content-Transfer-Encoding: 7bit -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - -Hi All, - - -Since I last posted to this list I have done some work -on a multi-threaded port of Postgres 7.0.2 that I have been kicking -around for a while. There has been some mild interest -in this in the past so I thought I might try and start a sourceforge -project with what I have so far. - ->From past discussions, it is clear to me that a direct port -of postgres which uses threads instead of processes is not a -good idea, how about an embedded version that uses threads. -A multi-threaded postgres might be good for that. -The version I am working on is slower in terms of transaction -throughput than the current postgres but it uses less system -resources and does not require shared memory. - -I know it is possible to embed the current postgres but I -believe that is a single user system. - -Comments? - - -Myron Scott -mkscott@sacadia.com - - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate -subscribe-nomail command to majordomo@postgresql.org so that your -message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M33671@postgresql.org Fri Jan 3 10:27:00 2003 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h03FQwl07124 - for ; Fri, 3 Jan 2003 10:26:58 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id EDEBC4764DE; Fri, 3 Jan 2003 10:26:53 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 31554476422; Fri, 3 Jan 2003 10:25:46 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id 69252476286 - for ; Fri, 3 Jan 2003 10:25:29 -0500 (EST) -Received: from www.pspl.co.in (www.pspl.co.in [202.54.11.65]) - by postgresql.org (Postfix) with ESMTP id 98F754764C3 - for ; Fri, 3 Jan 2003 10:23:52 -0500 (EST) -Received: (from root@localhost) - by www.pspl.co.in (8.11.6/8.11.6) id h03FNtK17518 - for ; Fri, 3 Jan 2003 20:53:55 +0530 -Received: from daithan.itnranet.pspl.co.in (daithan.intranet.pspl.co.in [192.168.7.161]) - by www.pspl.co.in (8.11.6/8.11.0) with ESMTP id h03FNsf17512 - for ; Fri, 3 Jan 2003 20:53:54 +0530 -From: Shridhar Daithankar -To: PGHackers -Subject: [HACKERS] Threads -Date: Fri, 3 Jan 2003 20:54:11 +0530 -User-Agent: KMail/1.4.3 -MIME-Version: 1.0 -Content-Type: Multipart/Mixed; - boundary="------------Boundary-00=_BG9530ZI94UNRKSGBVL5" -Message-ID: <200301032054.11125.shridhar_daithankar@persistent.co.in> -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Status: OR - ---------------Boundary-00=_BG9530ZI94UNRKSGBVL5 -Content-Type: text/plain; - charset="us-ascii" -Content-Transfer-Encoding: quoted-printable - -Hi all, - -I am sure, many of you would like to delete this message before reading, ho= -ld=20 -on. :-) - -There is much talk about threading on this list and the idea is always=20 -deferred for want of robust thread models across all supported platforms an= -d=20 -feasibility of gains v/s efforts required. - -I think threads are useful in difference situations namely parallelising=20 -blocking conditions and using multiple CPUs. - -Attached is a framework that I ported to C from a C++ server I have written= -.=20 -It has threadpool and threads implementation based on pthreads. - -This code expects minimum pthreads implementation and does not assume anyth= -ing=20 -on threads part (e.g kernel threads or not etc.) - -I request hackers on this list to take a look at it. It should be easily=20 -pluggable in any source code and is released without any strings for any us= -e. - -This framework allows to plug-in the worker function and argument on the fl= -y.=20 -The threads created are sleeping by default and can be woken up s and when= -=20 -required. - -I propose to use it incrementally in postgresql. Let's start with I/O. When= - a=20 -block of data is being read, rather than blocking for read, we can set up= -=20 -creator-consumer link between two threads That we way can utilize that I/O= -=20 -time in a overlapped fashion. - -Further threads can be useful when the server has more CPUs. It can spread = -CPU=20 -intensive work to different threads such as index creation or sorting. This= -=20 -way we can utilise idle CPU which we can not as of now. - -There are many advantages that I can see. - -1)Threads can be optionally turned on/off depending upon the configuration.= - So=20 -we can entirely keep existing functionality and convert them one-by-one to= -=20 -threaded application. - -2)For each functionality we can have two code branches, one that do not use= -=20 -threads i.e. current code base and one that can use threads. Agreed the=20 -binary will be bit bloated but that would give enormous flexibility. If we= -=20 -find a thread implementation buggy, we simply switch it off either in=20 -compilation or inconfiguration. - -3) Not much efforts should be required to plug code into this model. The id= -ea=20 -of using threads is to assign exclusive work to each thread. So that should= -=20 -not require much of a locking. - -In case of using multiple CPUs, separate functions need be written that can= -=20 -handle the things in a thread-safe fashion. Also a merger function would be= -=20 -required which would merge results of worker threads. That would be totally= -=20 -additional. - -I would say two threads per CPU per back-end should be a reasonable default= - as=20 -that would cover I/O blocking well. Of course unless threading is turned of= -f=20 -in build or in configuration. - -Please note that I have tested the code in C++ and my C is rusty. Quite lik= -ely=20 -there are bugs in the code. I will stress test the code on monday but I wou= -ld=20 -like to seek an opinion on this as soon as possible. ( Hey but it compiles= -=20 -clean..) - -If required I can post example usage of this code, but I don't think that= -=20 -should be necessary.:-) - -Bye - Shridhar - ---------------Boundary-00=_BG9530ZI94UNRKSGBVL5 -Content-Type: text/x-chdr; - charset="us-ascii"; - name="thread.h" -Content-Transfer-Encoding: 7bit -Content-Disposition: attachment; filename="thread.h" - -#define _REENTRANT - -#include -#include -#include -#include - - -//typedefs -typedef void* (*function)(void *); -typedef void* argtype; - -typedef struct -{ - pthread_mutex_t lock; - pthread_cond_t cond; - - unsigned short freeCount,n,count; - void *pool; - -} threadPool; - -typedef struct -{ - pthread_t t; - pthread_attr_t tattr; - pthread_mutex_t lock; - pthread_cond_t cond; - - argtype arg; - function f; - - unsigned short quit; - threadPool *p; - -} thread; - -/*Thread functions*/ -void initThread(thread **t,threadPool *pool); -void deleteThread(thread **t); -void stop(thread *thr); - -void wakeForWork(thread *thr,function func,argtype a); - -argtype runner(void *ptr); - -/*thread pool functions*/ -void initPool(threadPool **pool,unsigned short numthreads); -void deletePool(threadPool **p); - -void putThread(threadPool *p,thread *t); -thread *getThread(threadPool *p); - - - - - - ---------------Boundary-00=_BG9530ZI94UNRKSGBVL5 -Content-Type: text/x-csrc; - charset="us-ascii"; - name="thread.c" -Content-Transfer-Encoding: 7bit -Content-Disposition: attachment; filename="thread.c" - -#include "thread.h" - -void initThread(thread **t,threadPool *pool) -{ - thread *thr=(thread *)malloc(sizeof(thread)); - - if(!thr) - { - fprintf(stderr,"\nCan not allocate memory for thread. Quitting...\n"); - exit(1); - } - - *t=thr; - - pthread_attr_init(&(thr->tattr)); - pthread_mutex_init(&(thr->lock), NULL); - pthread_cond_init(&(thr->cond), NULL); - - pthread_attr_setdetachstate(&(thr->tattr),PTHREAD_CREATE_DETACHED); - - thr->quit=0; - thr->p=pool; - - //Create the thread - int ret=pthread_create(&(thr->t),&(thr->tattr),runner,(void *)thr); - - if(ret!=0) - { - fprintf(stderr,"\nCan not create thread. Quitting...\n"); - exit(1); - } -} - -void deleteThread(thread **t) -{ - thread *thr=*t; - - if(!t) return; - - stop(thr); - - pthread_attr_destroy(&(thr->tattr)); - pthread_cond_destroy(&(thr->cond)); - pthread_mutex_destroy(&(thr->lock)); - - free(thr); -} - -void stop(thread *thr) -{ - unsigned short i; - thr->quit=1; - - pthread_cond_signal(&(thr->cond)); - - for(i=0;thr->quit && i<10;i++) - { - if(i>=10) - { - pthread_kill(thr->t,9); - break; - } - usleep(400); - } -} - -void wakeForWork(thread *thr,function func,argtype a) -{ - thr->f=func; - thr->arg=a; - - pthread_cond_signal(&(thr->cond)); -} - -argtype runner(void* arg) -{ - thread *ptr=(thread *)arg; - - while(1) - { - pthread_mutex_lock(&(ptr->lock)); - - if(ptr->p) - putThread(ptr->p,ptr); - - pthread_cond_wait(&(ptr->cond),&(ptr->lock)); - - if(ptr->quit) break; - - ptr->f((void *)ptr->arg); - - pthread_mutex_unlock(&(ptr->lock)); - } - - ptr->quit=0; - - return NULL; -} - - -void initPool(threadPool **pool,unsigned short numthreads) -{ - thread **thr; - threadPool *p=(threadPool *)malloc(sizeof(threadPool)); - - if(!p) - { - fprintf(stderr,"Can not get memory to create threadpool. Quitting\n"); - exit(1); - } - - if(!pool) - { - free(p); - return; - } - - *pool=p; - - pthread_mutex_init(&(p->lock), NULL); - pthread_cond_init(&(p->cond), NULL); - - p->n=numthreads; - p->freeCount=0; - p->n=numthreads; - - thr=(thread **)malloc(numthreads*sizeof(thread *)); - - if(!thr) - { - fprintf(stderr,"Can not get memory to create pool of threads. Quitting\n"); - exit(1); - } - - p->pool=(void *)thr; - -} - -void deletePool(threadPool **pool) -{ - threadPool *p=(threadPool *)pool; - - if(!pool) return; - - thread **thr=(thread **)p->pool; - unsigned short i; - - for(i=0;in;i++) stop(thr[i]); - - free(p->pool); - - pthread_cond_destroy(&(p->cond)); - pthread_mutex_destroy(&(p->lock)); - - free(p); - -} - -void putThread(threadPool *p,thread *t) -{ - unsigned short i; - thread **pool; - - if(!p || !t) return; - - pool=(thread **)p->pool; - - pthread_mutex_lock(&(p->lock)); - - i=p->freeCount; - pool[(p->freeCount)++]=t; - - if(i<=0)pthread_cond_signal(&(p->cond)); - - pthread_mutex_unlock(&(p->lock)); - -} - -thread *getThread(threadPool *p) -{ - thread *t,**t1; - - if(!p) return NULL; - - t1=(thread **)p->pool; - - pthread_mutex_lock(&(p->lock)); - - if((p->freeCount)<=0)pthread_cond_wait(&(p->cond),&(p->lock)); - - t=t1[--(p->freeCount)]; - - pthread_mutex_unlock(&(p->lock)); - - return t; - -} - ---------------Boundary-00=_BG9530ZI94UNRKSGBVL5 -Content-Type: text/plain -Content-Disposition: inline -Content-Transfer-Encoding: 8bit -MIME-Version: 1.0 - - ----------------------------(end of broadcast)--------------------------- -TIP 6: Have you searched our list archives? - -http://archives.postgresql.org - ---------------Boundary-00=_BG9530ZI94UNRKSGBVL5-- - - -From pgsql-hackers-owner+M33682@postgresql.org Fri Jan 3 15:43:54 2003 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h03Khhl06938 - for ; Fri, 3 Jan 2003 15:43:45 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id DF70F476EA6; Fri, 3 Jan 2003 15:43:34 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 95BA8476514; Fri, 3 Jan 2003 15:43:26 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id 71F4E475DBC - for ; Fri, 3 Jan 2003 15:43:14 -0500 (EST) -Received: from snoopy.mohawksoft.com (h0030f1382639.ne.client2.attbi.com [24.60.194.163]) - by postgresql.org (Postfix) with ESMTP id ACE5B475DAD - for ; Fri, 3 Jan 2003 15:43:13 -0500 (EST) -Received: from mohawksoft.com (snoopy.mohawksoft.com [127.0.0.1]) - by snoopy.mohawksoft.com (8.11.6/8.11.6) with ESMTP id h03KlMs24421; - Fri, 3 Jan 2003 15:47:27 -0500 -Message-ID: <3E15F6DA.8000209@mohawksoft.com> -Date: Fri, 03 Jan 2003 15:47:22 -0500 -From: mlw -User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.1) Gecko/20020823 Netscape/7.0 -X-Accept-Language: en-us, en -MIME-Version: 1.0 -To: Shridhar Daithankar -cc: PGHackers -Subject: Re: [HACKERS] Threads -References: <200301032054.11125.shridhar_daithankar@persistent.co.in> -Content-Type: text/plain; charset=us-ascii; format=flowed -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Status: OR - -Please no threading threads!!! - -Has anyone calculated the interval and period of "PostgreSQL needs -threads" posts? - -The *ONLY* advantage threading has over multiple processes is the time -and resources used in creating new processes. - -That being said, I admit that creating a threaded program is easier than -one with multiple processes, but PostgreSQL is already there and working. - -Drawbacks to a threaded model: - -(1) One thread screws up, the whole process dies. In a multiple process -application this is not too much of an issue. - -(2) Heap fragmentation. In a long uptime application, such as a -database, heap fragmentation is an important consideration. With -multiple processes, each process manages its own heap and what ever -fragmentation that exists goes away when the connection is closed. A -threaded server is far more vulnerable because the heap has to manage -many threads and the heap has to stay active and unfragmented in -perpetuity. This is why Windows applications usually end up using 2G of -memory after 3 months of use. (Well, this AND memory leaks) - -(3) Stack space. In a threaded application they are more limits to stack -usage. I'm not sure, but I bet PostgreSQL would have a problem with a -fixed size stack, I know the old ODBC driver did. - -(4) Lock Contention. The various single points of access in a process -have to be serialized for multiple threads. heap allocation, -deallocation, etc all have to be managed. In a multple process model, -these resources would be separated by process contexts. - -(5) Lastly, why bother? Seriously? Process creation time is an issue -true, but its an issue with threads as well, just not as bad. Anyone who -is looking for performance should be using a connection pooling -mechanism as is done in things like PHP. - -I have done both threaded and process servers. The threaded servers are -easier to write. The process based severs are more robust. From an -operational point of view, a "select foo from bar where x > y" will take -he same amount of time. - - - - ----------------------------(end of broadcast)--------------------------- -TIP 6: Have you searched our list archives? - -http://archives.postgresql.org - -From pgsql-hackers-owner+M33684@postgresql.org Fri Jan 3 15:56:48 2003 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h03Kufl08003 - for ; Fri, 3 Jan 2003 15:56:43 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id D0392477118; Fri, 3 Jan 2003 15:56:31 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 31FDC475461; Fri, 3 Jan 2003 15:55:26 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id BD892477147 - for ; Fri, 3 Jan 2003 15:55:09 -0500 (EST) -Received: from voyager.corporate.connx.com (unknown [209.20.248.131]) - by postgresql.org (Postfix) with ESMTP id 7EE644771A0 - for ; Fri, 3 Jan 2003 15:52:47 -0500 (EST) -content-class: urn:content-classes:message -Subject: Re: [HACKERS] Threads -MIME-Version: 1.0 -Content-Type: text/plain; - charset="us-ascii" -Date: Fri, 3 Jan 2003 12:52:48 -0800 -X-MimeOLE: Produced By Microsoft Exchange V6.0.6249.0 -Message-ID: -Thread-Topic: [HACKERS] Threads -Thread-Index: AcKzaMsucwBFaOikSjKML8BqvR/gCAAACDPA -From: "Dann Corbit" -To: "PGHackers" -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Content-Transfer-Encoding: 8bit -X-MIME-Autoconverted: from quoted-printable to 8bit by candle.pha.pa.us id h03Kufl08003 -Status: OR - -> -----Original Message----- -> From: mlw [mailto:pgsql@mohawksoft.com] -> Sent: Friday, January 03, 2003 12:47 PM -> To: Shridhar Daithankar -> Cc: PGHackers -> Subject: Re: [HACKERS] Threads -> -> -> Please no threading threads!!! -> -> Has anyone calculated the interval and period of "PostgreSQL needs -> threads" posts? -> -> The *ONLY* advantage threading has over multiple processes is -> the time -> and resources used in creating new processes. - -Threading is absurdly easier to do portably than fork(). - -Will you fork() successfully on MVS, VMS, OS/2, Win32? - -On some operating systems, thread creation is absurdly faster than -process creation (many orders of magnitude). - -> That being said, I admit that creating a threaded program is -> easier than -> one with multiple processes, but PostgreSQL is already there -> and working. -> -> Drawbacks to a threaded model: -> -> (1) One thread screws up, the whole process dies. In a -> multiple process -> application this is not too much of an issue. - -If you use C++ you can try/catch and nothing bad happens to anything but -the naughty thread. - -> (2) Heap fragmentation. In a long uptime application, such as a -> database, heap fragmentation is an important consideration. With -> multiple processes, each process manages its own heap and what ever -> fragmentation that exists goes away when the connection is closed. A -> threaded server is far more vulnerable because the heap has to manage -> many threads and the heap has to stay active and unfragmented in -> perpetuity. This is why Windows applications usually end up -> using 2G of -> memory after 3 months of use. (Well, this AND memory leaks) - -Poorly written applications leak memory. Fragmentation is a legitimate -concern. - -> (3) Stack space. In a threaded application they are more -> limits to stack -> usage. I'm not sure, but I bet PostgreSQL would have a problem with a -> fixed size stack, I know the old ODBC driver did. - -A single server with 20 threads will consume less total free store -memory and automatic memory than 20 servers. You have to decide how -much stack to give a thread, that's true. - -> (4) Lock Contention. The various single points of access in a process -> have to be serialized for multiple threads. heap allocation, -> deallocation, etc all have to be managed. In a multple process model, -> these resources would be separated by process contexts. - -Semaphores are more complicated than critical sections. If anything, a -shared memory approach is more problematic and fragile, especially when -porting to multiple operating systems. - -> (5) Lastly, why bother? Seriously? Process creation time is an issue -> true, but its an issue with threads as well, just not as bad. -> Anyone who -> is looking for performance should be using a connection pooling -> mechanism as is done in things like PHP. -> -> I have done both threaded and process servers. The threaded -> servers are -> easier to write. The process based severs are more robust. From an -> operational point of view, a "select foo from bar where x > -> y" will take -> he same amount of time. - -Probably true. I think a better solution is a server that can start -threads or processes or both. But that's neither here nor there and I'm -certainly not volunteering to write it. - -Here is a solution to the dilemma. Make the one who suggests the -feature be the first volunteer on the team that writes it. - -Is it a FAQ? If not, it ought to be. - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate -subscribe-nomail command to majordomo@postgresql.org so that your -message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M33685@postgresql.org Fri Jan 3 16:35:02 2003 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h03LYsl11402 - for ; Fri, 3 Jan 2003 16:34:56 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id 0F09B477168; Fri, 3 Jan 2003 16:34:48 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id C1A9C477132; Fri, 3 Jan 2003 16:34:39 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id D830847630B - for ; Fri, 3 Jan 2003 16:34:25 -0500 (EST) -Received: from CopelandConsulting.Net (dsl-24293-ld.customer.centurytel.net [209.142.135.135]) - by postgresql.org (Postfix) with ESMTP id 025DD476417 - for ; Fri, 3 Jan 2003 16:34:24 -0500 (EST) -Received: from [192.168.1.2] (mouse.copelandconsulting.net [192.168.1.2]) - by CopelandConsulting.Net (8.10.1/8.10.1) with ESMTP id h03LY2700731; - Fri, 3 Jan 2003 15:34:03 -0600 (CST) -X-Trade-Id: -To: mlw -cc: Shridhar Daithankar , - PGHackers -In-Reply-To: <3E15F6DA.8000209@mohawksoft.com> -References: <200301032054.11125.shridhar_daithankar@persistent.co.in> - <3E15F6DA.8000209@mohawksoft.com> -Content-Type: text/plain -Organization: Copeland Computer Consulting -Message-ID: <1041629649.15933.135.camel@mouse.copelandconsulting.net> -MIME-Version: 1.0 -X-Mailer: Ximian Evolution 1.2.0 -Date: 03 Jan 2003 15:34:10 -0600 -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Status: OR - -On Fri, 2003-01-03 at 14:47, mlw wrote: -> Please no threading threads!!! -> - -Ya, I'm very pro threads but I've long since been sold on no threads for -PostgreSQL. AIO on the other hand... ;) - -Your summary so accurately addresses the issue it should be a whole FAQ -entry on threads and PostgreSQL. :) - - -> Drawbacks to a threaded model: -> -> (1) One thread screws up, the whole process dies. In a multiple process -> application this is not too much of an issue. -> -> (2) Heap fragmentation. In a long uptime application, such as a -> database, heap fragmentation is an important consideration. With -> multiple processes, each process manages its own heap and what ever -> fragmentation that exists goes away when the connection is closed. A -> threaded server is far more vulnerable because the heap has to manage -> many threads and the heap has to stay active and unfragmented in -> perpetuity. This is why Windows applications usually end up using 2G of -> memory after 3 months of use. (Well, this AND memory leaks) - - -These are things that can't be stressed enough. IMO, these are some of -the many reasons why applications running on MS platforms tend to have -much lower application and system up times (that and resources leaks -which are inherent to the platform). - -BTW, if you do much in the way of threaded coding, there is libHorde -which is a heap library for heavily threaded, memory hungry -applications. It excels in performance, reduces heap lock contention -(maintains multiple heaps in a very thread smart manner), and goes a -long way toward reducing heap fragmentation which is common for heavily -memory based, threaded applications. - - -> (3) Stack space. In a threaded application they are more limits to stack -> usage. I'm not sure, but I bet PostgreSQL would have a problem with a -> fixed size stack, I know the old ODBC driver did. -> - -Most modern thread implementations use a page guard on the stack to -determine if it needs to grow or not. Generally speaking, for most -modern platforms which support threading, stack considerations rarely -become an issue. - - -> (5) Lastly, why bother? Seriously? Process creation time is an issue -> true, but its an issue with threads as well, just not as bad. Anyone who -> is looking for performance should be using a connection pooling -> mechanism as is done in things like PHP. -> -> I have done both threaded and process servers. The threaded servers are -> easier to write. The process based severs are more robust. From an -> operational point of view, a "select foo from bar where x > y" will take -> he same amount of time. -> - -I agree with this, however, using threads does open the door for things -like splitting queries and sorts across multiple CPUs. Something the -current process model, which was previously agreed on, would not be able -to address because of cost. - -Example: "select foo from bar where x > y order by foo ;", could be run -on multiple CPUs if the sort were large enough to justify. - -After it's all said and done, I do agree that threading just doesn't -seem like a good fit for PostgreSQL. - --- -Greg Copeland -Copeland Computer Consulting - - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M33686@postgresql.org Fri Jan 3 16:47:20 2003 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h03LlBl12502 - for ; Fri, 3 Jan 2003 16:47:12 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id 6873147621D; Fri, 3 Jan 2003 16:47:06 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 97466477133; Fri, 3 Jan 2003 16:46:41 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id E25BB477152 - for ; Fri, 3 Jan 2003 16:46:24 -0500 (EST) -Received: from CopelandConsulting.Net (dsl-24293-ld.customer.centurytel.net [209.142.135.135]) - by postgresql.org (Postfix) with ESMTP id 84A87477157 - for ; Fri, 3 Jan 2003 16:45:21 -0500 (EST) -Received: from [192.168.1.2] (mouse.copelandconsulting.net [192.168.1.2]) - by CopelandConsulting.Net (8.10.1/8.10.1) with ESMTP id h03LjC712426; - Fri, 3 Jan 2003 15:45:13 -0600 (CST) -X-Trade-Id: -To: Dann Corbit -cc: PGHackers -In-Reply-To: -References: -Content-Type: text/plain -Organization: Copeland Computer Consulting -Message-ID: <1041630319.15927.146.camel@mouse.copelandconsulting.net> -MIME-Version: 1.0 -X-Mailer: Ximian Evolution 1.2.0 -Date: 03 Jan 2003 15:45:20 -0600 -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Status: OR - -On Fri, 2003-01-03 at 14:52, Dann Corbit wrote: -> > -----Original Message----- -> > (1) One thread screws up, the whole process dies. In a -> > multiple process -> > application this is not too much of an issue. -> -> If you use C++ you can try/catch and nothing bad happens to anything but -> the naughty thread. - -That doesn't protect against the type of issues he's talking about. -Invalid pointer reference is a very common snafu which really hoses -threaded applications. Not to mention resource leaks AND LOCKED -resources which are inherently an issue on Win32. - -Besides, it's doubtful that PostgreSQL is going to be rewritten in C++ -so bringing up try/catch is pretty much an invalid argument. - -> -> > (2) Heap fragmentation. In a long uptime application, such as a -> > database, heap fragmentation is an important consideration. With -> > multiple processes, each process manages its own heap and what ever -> > fragmentation that exists goes away when the connection is closed. A -> > threaded server is far more vulnerable because the heap has to manage -> > many threads and the heap has to stay active and unfragmented in -> > perpetuity. This is why Windows applications usually end up -> > using 2G of -> > memory after 3 months of use. (Well, this AND memory leaks) -> -> Poorly written applications leak memory. Fragmentation is a legitimate -> concern. - -And well written applications which attempt to safely handle segfaults, -etc., often leak memory and lock resources like crazy. On Win32, -depending on the nature of the resources, once this happens, even -process termination will not free/unlock the resources. - -> > (4) Lock Contention. The various single points of access in a process -> > have to be serialized for multiple threads. heap allocation, -> > deallocation, etc all have to be managed. In a multple process model, -> > these resources would be separated by process contexts. -> -> Semaphores are more complicated than critical sections. If anything, a -> shared memory approach is more problematic and fragile, especially when -> porting to multiple operating systems. - -And critical sections lead to low performance on SMP systems for Win32 -platforms. No task can switch on ANY CPU for the duration of the -critical section. It's highly recommend by MS as the majority of Win32 -applications expect uniprocessor systems and they are VERY fast. As -soon as multiple processors come into the mix, critical sections become -a HORRIBLE idea if any soft of scalability is desired. - - -> Is it a FAQ? If not, it ought to be. - -I agree. I think mlw's list of reasons should be added to a faq. It -terse yet says it all! - - --- -Greg Copeland -Copeland Computer Consulting - - ----------------------------(end of broadcast)--------------------------- -TIP 2: you can get off all lists at once with the unregister command - (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) - -From pgsql-hackers-owner+M33703@postgresql.org Fri Jan 3 20:41:10 2003 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h041f9l05824 - for ; Fri, 3 Jan 2003 20:41:09 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id 7F5764764C8; Fri, 3 Jan 2003 20:41:04 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id BE24547606D; Fri, 3 Jan 2003 20:38:53 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id 4D50D476165 - for ; Fri, 3 Jan 2003 20:38:39 -0500 (EST) -Received: from sss.pgh.pa.us (unknown [192.204.191.242]) - by postgresql.org (Postfix) with ESMTP id 20C8547659F - for ; Fri, 3 Jan 2003 20:34:10 -0500 (EST) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.12.6/8.12.6) with ESMTP id h041Y20U023764; - Fri, 3 Jan 2003 20:34:03 -0500 (EST) -To: "Serguei Mokhov" -cc: "Greg Copeland" , - "Dann Corbit" , - "PGHackers" -Subject: Re: [HACKERS] Threads -In-Reply-To: <004101c2b37b$0f261ae0$0301a8c0@gunnymede.lan> -References: <1041630319.15927.146.camel@mouse.copelandconsulting.net> <004101c2b37b$0f261ae0$0301a8c0@gunnymede.lan> -Comments: In-reply-to "Serguei Mokhov" - message dated "Fri, 03 Jan 2003 17:54:20 -0500" -Date: Fri, 03 Jan 2003 20:34:02 -0500 -Message-ID: <23763.1041644042@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Status: OR - -"Serguei Mokhov" writes: ->>> (1) One thread screws up, the whole process dies. In a ->>> multiple process application this is not too much of an issue. - -> (1) is an issue only for user-level threads. - -Uh, what other kind of thread have you got in mind here? - -I suppose the lack-of-cross-thread-protection issue would go away if -our objective was only to use threads for internal parallelism in each -backend instance (ie, you still have one process per connection, but -internally it would use multiple threads to process subqueries in -parallel). - -Of course that gives up the hope of faster connection startup that has -always been touted as a major reason to want Postgres to be threaded... - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 5: Have you checked our extensive FAQ? - -http://www.postgresql.org/users-lounge/docs/faq.html - -From pgsql-hackers-owner+M33706@postgresql.org Fri Jan 3 21:16:55 2003 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h042Gsl08584 - for ; Fri, 3 Jan 2003 21:16:54 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id 8F2EB475E22; Fri, 3 Jan 2003 21:16:49 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 72017475FDA; Fri, 3 Jan 2003 21:15:21 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id EA790476242 - for ; Fri, 3 Jan 2003 21:15:00 -0500 (EST) -Received: from CopelandConsulting.Net (dsl-24293-ld.customer.centurytel.net [209.142.135.135]) - by postgresql.org (Postfix) with ESMTP id BB7A0475D0D - for ; Fri, 3 Jan 2003 21:11:20 -0500 (EST) -Received: from [192.168.1.2] (mouse.copelandconsulting.net [192.168.1.2]) - by CopelandConsulting.Net (8.10.1/8.10.1) with ESMTP id h042B8729407; - Fri, 3 Jan 2003 20:11:08 -0600 (CST) -X-Trade-Id: -To: Tom Lane -cc: Serguei Mokhov , Dann Corbit , - PGHackers -In-Reply-To: <23763.1041644042@sss.pgh.pa.us> -References: - <1041630319.15927.146.camel@mouse.copelandconsulting.net> - <004101c2b37b$0f261ae0$0301a8c0@gunnymede.lan> - <23763.1041644042@sss.pgh.pa.us> -Content-Type: text/plain -Organization: Copeland Computer Consulting -Message-ID: <1041646276.15927.202.camel@mouse.copelandconsulting.net> -MIME-Version: 1.0 -X-Mailer: Ximian Evolution 1.2.0 -Date: 03 Jan 2003 20:11:17 -0600 -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Status: OR - -On Fri, 2003-01-03 at 19:34, Tom Lane wrote: -> "Serguei Mokhov" writes: -> >>> (1) One thread screws up, the whole process dies. In a -> >>> multiple process application this is not too much of an issue. -> -> > (1) is an issue only for user-level threads. -> - - -Umm. No. User or system level threads, the statement is true. If a -thread kills over, the process goes with it. Furthermore, on Win32 -platforms, it opens a whole can of worms no matter how you care to -address it. - -> Uh, what other kind of thread have you got in mind here? -> -> I suppose the lack-of-cross-thread-protection issue would go away if -> our objective was only to use threads for internal parallelism in each -> backend instance (ie, you still have one process per connection, but -> internally it would use multiple threads to process subqueries in -> parallel). -> - -Several have previously spoken about a hybrid approach (ala Apache). -IIRC, it was never ruled out but it was simply stated that no one had -the energy to put into such a concept. - -> Of course that gives up the hope of faster connection startup that has -> always been touted as a major reason to want Postgres to be threaded... -> -> regards, tom lane - -Faster startup, should never be the primary reason as there are many -ways to address that issue already. Connection pooling and caching are -by far, the most common way to address this issue. Not only that, but -by definition, it's almost an oxymoron. If you really need high -performance, you shouldn't be using transient connections, no matter how -fast they are. This, in turn, brings you back to persistent connections -or connection pools/caches. - - --- -Greg Copeland -Copeland Computer Consulting - - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M33709@postgresql.org Fri Jan 3 22:39:26 2003 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h043dOl13614 - for ; Fri, 3 Jan 2003 22:39:25 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id CA13B47621C; Fri, 3 Jan 2003 22:39:20 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 8DE1D475DFF; Fri, 3 Jan 2003 22:39:04 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id 15AA1475AFF - for ; Fri, 3 Jan 2003 22:39:00 -0500 (EST) -Received: from CopelandConsulting.Net (dsl-24293-ld.customer.centurytel.net [209.142.135.135]) - by postgresql.org (Postfix) with ESMTP id 19D8F475ADD - for ; Fri, 3 Jan 2003 22:38:59 -0500 (EST) -Received: from [192.168.1.2] (mouse.copelandconsulting.net [192.168.1.2]) - by CopelandConsulting.Net (8.10.1/8.10.1) with ESMTP id h043ca714568; - Fri, 3 Jan 2003 21:38:36 -0600 (CST) -X-Trade-Id: -To: mlw -cc: Tom Lane , Serguei Mokhov , - Dann Corbit , PGHackers -In-Reply-To: <3E16575C.1030805@mohawksoft.com> -References: - <1041630319.15927.146.camel@mouse.copelandconsulting.net> - <004101c2b37b$0f261ae0$0301a8c0@gunnymede.lan> - <23763.1041644042@sss.pgh.pa.us> - <1041646276.15927.202.camel@mouse.copelandconsulting.net> - <3E16575C.1030805@mohawksoft.com> -Content-Type: text/plain -Organization: Copeland Computer Consulting -Message-ID: <1041651525.15927.207.camel@mouse.copelandconsulting.net> -MIME-Version: 1.0 -X-Mailer: Ximian Evolution 1.2.0 -Date: 03 Jan 2003 21:38:46 -0600 -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Status: OR - -On Fri, 2003-01-03 at 21:39, mlw wrote: -> Connection time should *never* be in the critical path. There, I've -> said it!! People who complain about connection time are barking up the -> wrong tree. Regardless of the methodology, EVERY OS has issues with -> thread creation, process creation, the memory allocation, and system -> manipulation required to manage it. Under load this is ALWAYS slower. -> -> I think that if there is ever a choice, "do I make startup time -> faster?" or "Do I make PostgreSQL not need a dump/restore for upgrade" -> the upgrade problem has a much higher impact to real PostgreSQL sites. - - -Exactly. Trying to speed up something that shouldn't be in the critical -path is exactly what I'm talking about. - -I completely agree with you! - - --- -Greg Copeland -Copeland Computer Consulting - - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M33708@postgresql.org Fri Jan 3 22:35:26 2003 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h043ZOl13418 - for ; Fri, 3 Jan 2003 22:35:25 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id 2277B475FDA; Fri, 3 Jan 2003 22:35:21 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id DA681475E18; Fri, 3 Jan 2003 22:35:12 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id 8254047595A - for ; Fri, 3 Jan 2003 22:34:58 -0500 (EST) -Received: from snoopy.mohawksoft.com (h0030f1382639.ne.client2.attbi.com [24.60.194.163]) - by postgresql.org (Postfix) with ESMTP id A4D60475921 - for ; Fri, 3 Jan 2003 22:34:57 -0500 (EST) -Received: from mohawksoft.com (snoopy.mohawksoft.com [127.0.0.1]) - by snoopy.mohawksoft.com (8.11.6/8.11.6) with ESMTP id h043d8s26180; - Fri, 3 Jan 2003 22:39:09 -0500 -Message-ID: <3E16575C.1030805@mohawksoft.com> -Date: Fri, 03 Jan 2003 22:39:08 -0500 -From: mlw -User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.1) Gecko/20020823 Netscape/7.0 -X-Accept-Language: en-us, en -MIME-Version: 1.0 -To: Greg Copeland -cc: Tom Lane , Serguei Mokhov , - Dann Corbit , PGHackers -Subject: Re: [HACKERS] Threads -References: <1041630319.15927.146.camel@mouse.copelandconsulting.net> <004101c2b37b$0f261ae0$0301a8c0@gunnymede.lan> <23763.1041644042@sss.pgh.pa.us> <1041646276.15927.202.camel@mouse.copelandconsulting.net> -Content-Type: multipart/alternative; - boundary="------------030005060103020905060907" -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Status: OR - ---------------030005060103020905060907 -Content-Type: text/plain; charset=us-ascii; format=flowed -Content-Transfer-Encoding: 7bit - - - -Greg Copeland wrote: - -> -> ->>Of course that gives up the hope of faster connection startup that has ->>always been touted as a major reason to want Postgres to be threaded... ->> ->> regards, tom lane ->> ->> -> ->Faster startup, should never be the primary reason as there are many ->ways to address that issue already. Connection pooling and caching are ->by far, the most common way to address this issue. Not only that, but ->by definition, it's almost an oxymoron. If you really need high ->performance, you shouldn't be using transient connections, no matter how ->fast they are. This, in turn, brings you back to persistent connections ->or connection pools/caches. -> -Connection time should *never* be in the critical path. There, I've said -it!! People who complain about connection time are barking up the wrong -tree. Regardless of the methodology, EVERY OS has issues with thread -creation, process creation, the memory allocation, and system -manipulation required to manage it. Under load this is ALWAYS slower. - -I think that if there is ever a choice, "do I make startup time faster?" -or "Do I make PostgreSQL not need a dump/restore for upgrade" the -upgrade problem has a much higher impact to real PostgreSQL sites. - ---------------030005060103020905060907 -Content-Type: text/html; charset=us-ascii -Content-Transfer-Encoding: 7bit - - - - - - - -
-
-Greg Copeland wrote:
-
- 
-
-
- 
Of course that gives up the hope of faster connection startup that has
-always been touted as a major reason to want Postgres to be threaded...
-
-			regards, tom lane
-
-
- 

-Faster startup, should never be the primary reason as there are many
-ways to address that issue already.  Connection pooling and caching are
-by far, the most common way to address this issue.  Not only that, but
-by definition, it's almost an oxymoron.  If you really need high
-performance, you shouldn't be using transient connections, no matter how
-fast they are.  This, in turn, brings you back to persistent connections
-or connection pools/caches.
-
-Connection time should *never* be in the critical path. There, I've said -it!! People who complain about connection time are barking up the wrong tree. -Regardless of the methodology, EVERY OS has issues with thread creation, -process creation, the memory allocation, and system manipulation  required -to manage it. Under load this is ALWAYS slower.
-
-I think that if there is ever a choice, "do I make startup time faster?" -or "Do I make PostgreSQL not need a dump/restore for upgrade" the upgrade -problem has a much higher impact to real PostgreSQL sites.
- - - ---------------030005060103020905060907-- - - -From pgsql-hackers-owner+M33713@postgresql.org Sat Jan 4 00:34:04 2003 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h045Y2l23520 - for ; Sat, 4 Jan 2003 00:34:02 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id BCA39476226; Sat, 4 Jan 2003 00:33:56 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 1B030475F09; Sat, 4 Jan 2003 00:33:47 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id A42D847595A - for ; Sat, 4 Jan 2003 00:33:37 -0500 (EST) -Received: from houston.familyhealth.com.au (unknown [203.59.48.253]) - by postgresql.org (Postfix) with ESMTP id C14B4475921 - for ; Sat, 4 Jan 2003 00:33:35 -0500 (EST) -Received: from localhost (chriskl@localhost) - by houston.familyhealth.com.au (8.11.6/8.11.6) with ESMTP id h045XKt36362; - Sat, 4 Jan 2003 13:33:23 +0800 (WST) - (envelope-from chriskl@familyhealth.com.au) -Date: Sat, 4 Jan 2003 13:33:20 +0800 (WST) -From: Christopher Kings-Lynne -To: mlw -cc: Shridhar Daithankar , - PGHackers -Subject: Re: [HACKERS] Threads -In-Reply-To: <3E15F6DA.8000209@mohawksoft.com> -Message-ID: <20030104133226.N36192-100000@houston.familyhealth.com.au> -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Status: OR - -Also remember that in even well developed OS's like FreeBSD, all a -process's threads will execute only on one CPU. This might change in -FreeBSD 5.0, but still a threaded app (such as MySQL) cannot use mutliple -CPUs on a FreeBSD system. - -Chris - -On Fri, 3 Jan 2003, mlw wrote: - -> Please no threading threads!!! -> -> Has anyone calculated the interval and period of "PostgreSQL needs -> threads" posts? -> -> The *ONLY* advantage threading has over multiple processes is the time -> and resources used in creating new processes. -> -> That being said, I admit that creating a threaded program is easier than -> one with multiple processes, but PostgreSQL is already there and working. -> -> Drawbacks to a threaded model: -> -> (1) One thread screws up, the whole process dies. In a multiple process -> application this is not too much of an issue. -> -> (2) Heap fragmentation. In a long uptime application, such as a -> database, heap fragmentation is an important consideration. With -> multiple processes, each process manages its own heap and what ever -> fragmentation that exists goes away when the connection is closed. A -> threaded server is far more vulnerable because the heap has to manage -> many threads and the heap has to stay active and unfragmented in -> perpetuity. This is why Windows applications usually end up using 2G of -> memory after 3 months of use. (Well, this AND memory leaks) -> -> (3) Stack space. In a threaded application they are more limits to stack -> usage. I'm not sure, but I bet PostgreSQL would have a problem with a -> fixed size stack, I know the old ODBC driver did. -> -> (4) Lock Contention. The various single points of access in a process -> have to be serialized for multiple threads. heap allocation, -> deallocation, etc all have to be managed. In a multple process model, -> these resources would be separated by process contexts. -> -> (5) Lastly, why bother? Seriously? Process creation time is an issue -> true, but its an issue with threads as well, just not as bad. Anyone who -> is looking for performance should be using a connection pooling -> mechanism as is done in things like PHP. -> -> I have done both threaded and process servers. The threaded servers are -> easier to write. The process based severs are more robust. From an -> operational point of view, a "select foo from bar where x > y" will take -> he same amount of time. -> -> -> -> -> ---------------------------(end of broadcast)--------------------------- -> TIP 6: Have you searched our list archives? -> -> http://archives.postgresql.org -> - - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate -subscribe-nomail command to majordomo@postgresql.org so that your -message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M33723@postgresql.org Sat Jan 4 13:21:52 2003 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h04ILpl25640 - for ; Sat, 4 Jan 2003 13:21:51 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id A5E5D4764F0; Sat, 4 Jan 2003 13:21:50 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id B8D94476021; Sat, 4 Jan 2003 13:21:37 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id 7FDFE475CE7 - for ; Sat, 4 Jan 2003 13:21:28 -0500 (EST) -Received: from CopelandConsulting.Net (dsl-24293-ld.customer.centurytel.net [209.142.135.135]) - by postgresql.org (Postfix) with ESMTP id 71C47474E42 - for ; Sat, 4 Jan 2003 13:21:27 -0500 (EST) -Received: from [192.168.1.2] (mouse.copelandconsulting.net [192.168.1.2]) - by CopelandConsulting.Net (8.10.1/8.10.1) with ESMTP id h04ILF721061; - Sat, 4 Jan 2003 12:21:15 -0600 (CST) -X-Trade-Id: -To: kar@kakidata.dk -cc: PGHackers -In-Reply-To: <200301041359.35715.kar@kakidata.dk> -References: - <23763.1041644042@sss.pgh.pa.us> - <1041646276.15927.202.camel@mouse.copelandconsulting.net> - <200301041359.35715.kar@kakidata.dk> -Content-Type: text/plain -Organization: Copeland Computer Consulting -Message-ID: <1041704480.15927.224.camel@mouse.copelandconsulting.net> -MIME-Version: 1.0 -X-Mailer: Ximian Evolution 1.2.0 -Date: 04 Jan 2003 12:21:20 -0600 -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Status: OR - -On Sat, 2003-01-04 at 06:59, Kaare Rasmussen wrote: -> > Umm. No. User or system level threads, the statement is true. If a -> > thread kills over, the process goes with it. Furthermore, on Win32 -> -> Hm. This is a database system. If one of the backend processes dies -> unexpectedly, I'm not sure I would trust the consistency and state of the -> others. -> -> Or maybe I'm just being chicken. - -I'd call that being wise. That's the problem with using threads. -Should a thread do something naughty, the state of the entire process is -in question. This is true regardless if it is a user mode, kernel mode, -or hybrid thread implementation. That's the power of using the process -model that is currently in use. Should it do something naughty, we -bitch and complain politely, throw our hands in the air and exit. We no -longer have to worry about the state and validity of that backend. This -creates a huge systemic reliability surplus. - -This is also why the concept of a hybrid thread/process implementation -keeps coming to the surface on the list. If you maintain the process -model and only use threads for things that ONLY relate to the single -process (single session/connection), should a thread cause a problem, -you can still throw you hands in the air and exit just as is done now -without causing problems for, or questioning the validity of, other -backends. - -The cool thing about such a concept is that it still opens the door for -things like parallel sorts and queries as it relates to a single -backend. - - --- -Greg Copeland -Copeland Computer Consulting - - ----------------------------(end of broadcast)--------------------------- -TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org - -From pgsql-hackers-owner+M33819@postgresql.org Mon Jan 6 02:41:01 2003 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h067exi23864 - for ; Mon, 6 Jan 2003 02:40:59 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id CCD564763B7; Mon, 6 Jan 2003 02:40:56 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 4A6574762E0; Mon, 6 Jan 2003 02:40:54 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id 2C31947606A - for ; Mon, 6 Jan 2003 02:40:50 -0500 (EST) -Received: from datafix.CS.Berkeley.EDU (datafix.CS.Berkeley.EDU [128.32.37.185]) - by postgresql.org (Postfix) with ESMTP id 8D7AF47603D - for ; Mon, 6 Jan 2003 02:40:49 -0500 (EST) -Received: (from sailesh@localhost) - by datafix.CS.Berkeley.EDU (8.11.6/8.11.6) id h067ac532006; - Sun, 5 Jan 2003 23:36:38 -0800 -X-Authentication-Warning: datafix.CS.Berkeley.EDU: sailesh set sender to sailesh@cs.berkeley.edu using -f -Reply-To: sailesh@cs.berkeley.edu -X-URL: http://www.cs.berkeley.edu/~sailesh -X-Attribution: Sailesh -To: Shridhar Daithankar -cc: PGHackers -Subject: Re: [HACKERS] Threads -References: <200301032054.11125.shridhar_daithankar@persistent.co.in> - <3E1605B8.5060403@priefert.com> - <200301061202.43247.shridhar_daithankar@persistent.co.in> -From: Sailesh Krishnamurthy -Date: 05 Jan 2003 23:36:38 -0800 -In-Reply-To: <200301061202.43247.shridhar_daithankar@persistent.co.in> -Message-ID: -Lines: 50 -User-Agent: Gnus/5.0808 (Gnus v5.8.8) XEmacs/21.1 (Cuyahoga Valley) -MIME-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Status: OR - ->>>>> "Shridhar" == Shridhar Daithankar writes: - - Shridhar> On Saturday 04 January 2003 03:20 am, you wrote: - >> >I am sure, many of you would like to delete this message - >> before reading, > hold on. :-) - >> - >> I'm afraid most posters did not read the message. Those who - >> replied - >> - >> "Why bother?" did not address your challenge: - - Shridhar> Our challenges may be..;-) - -Not having threading does reduce some of the freedom we've been having -in our work. But then we have ripped the process model a fair bit and -we have the freedom of an entirely new process to deal with data -streams entering the system and we're experimenting with threading for -asynchronous I/O there. - -However, in general I agree with the spirit of the previous messages -in this thread that threading isn't the main issue for PG. - -One thing that I missed so far in the threading thread. Context -switches are (IMHO) far cheaper between threads, because you save TLB -flushes. Whether this makes a real difference in a data intensive -application, I don't know. I wonder how easy it is to measure the x86 -counters to see TLB flushes/misses. - -In a database system, even if one process dies, I'd be very chary of -trusting it. So I am not too swayed by the fact that a -process-per-connection gets you better isolation. - -BTW, many commercial database systems also use per-process models on -Unix. However they are very aggressive with connection sharing and -reuse - even to the point of reusing the same process for multiple -active connections .. maybe at transaction boundaries. Good when a -connection is maintained for a long duaration with short-lived -transactions separated by fair amouns of time. - -Moreover, in db2 for instance, the same code base is used for both -per-thread and per-process models - in other words, the entire code is -MT-safe, and the scheduling mechanism is treated as a policy (Win32 is -MT, and some Unices MP). AFAICT though, postgres code, such as perhaps -the memory contexts is not MT-safe (of course the bufferpool/shmem -accesses are safe). - --- -Pip-pip -Sailesh -http://www.cs.berkeley.edu/~sailesh - ----------------------------(end of broadcast)--------------------------- -TIP 6: Have you searched our list archives? - -http://archives.postgresql.org - -From pgsql-hackers-owner+M33822@postgresql.org Mon Jan 6 06:23:29 2003 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h06BNSi17987 - for ; Mon, 6 Jan 2003 06:23:28 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id A1204476260; Mon, 6 Jan 2003 06:23:21 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 0B78D476060; Mon, 6 Jan 2003 06:23:19 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id 50277475BA0 - for ; Mon, 6 Jan 2003 06:23:14 -0500 (EST) -Received: from mail.gne.de (mail.gne.de [213.83.0.2]) - by postgresql.org (Postfix) with ESMTP id 27B244758E6 - for ; Mon, 6 Jan 2003 06:23:13 -0500 (EST) -Received: from DO5GNE-MTA by mail.gne.de - with Novell_GroupWise; Mon, 06 Jan 2003 12:23:02 +0100 -Message-ID: -X-Mailer: Novell GroupWise Internet Agent 6.0.2 -Date: Mon, 06 Jan 2003 12:22:57 +0100 -From: "Ulrich Neumann" -To: " -Subject: Re: [HACKERS] Threads -MIME-Version: 1.0 -Content-Type: text/plain; charset=US-ASCII -Content-Transfer-Encoding: 7bit -Content-Disposition: inline -X-Guinevere: 1.1.14 ; GNE Grebe Neumann Gl -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Status: OR - -Hello all, - -it's very interesting to see the discussion of "threads" again. - -I've portet PostgreSQL to a "thread-per-connection" model based on -pthreads -and it is functional. Most of the work was finding all the static -globals in the sourcefiles -and swapping them between threads and freeing memory if a thread -terminates. -(PostgreSQL isn't written very clean in the aspects of memory -handling). - -My version of the thread-based PostgreSQL is not very efficient at the -moment because -I haven't done any optimisation of the code to better support threads -and I'm using just a -simple semaphore to control switching of data but this could be a -starting point for -others who want to see this code. If this direction will be taken -seriously I'm very willing -to help. - -If someone is interested in the code I can send a zip file to everyone -who wants. - -Ulrich ----------------------------------- - This e-mail is virus scanned - Diese e-mail ist virusgeprueft - - ----------------------------(end of broadcast)--------------------------- -TIP 5: Have you checked our extensive FAQ? - -http://www.postgresql.org/users-lounge/docs/faq.html - -From pgsql-hackers-owner+M33824@postgresql.org Mon Jan 6 07:49:46 2003 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h06Cnii03541 - for ; Mon, 6 Jan 2003 07:49:44 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id C409E476778; Mon, 6 Jan 2003 07:49:36 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 617C04768C8; Mon, 6 Jan 2003 07:49:01 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id EA9284768AA - for ; Mon, 6 Jan 2003 07:48:56 -0500 (EST) -Received: from CopelandConsulting.Net (dsl-24293-ld.customer.centurytel.net [209.142.135.135]) - by postgresql.org (Postfix) with ESMTP id 2DB74476191 - for ; Mon, 6 Jan 2003 07:48:41 -0500 (EST) -Received: from [192.168.1.2] (mouse.copelandconsulting.net [192.168.1.2]) - by CopelandConsulting.Net (8.10.1/8.10.1) with ESMTP id h06CmL702059; - Mon, 6 Jan 2003 06:48:21 -0600 (CST) -X-Trade-Id: -To: shridhar_daithankar@persistent.co.in -cc: " -In-Reply-To: <3E19B78B.25689.15BFFE@localhost> -References: <3E19B78B.25689.15BFFE@localhost> -Content-Type: text/plain -Organization: Copeland Computer Consulting -Message-ID: <1041857302.17321.49.camel@mouse.copelandconsulting.net> -MIME-Version: 1.0 -X-Mailer: Ximian Evolution 1.2.0 -Date: 06 Jan 2003 06:48:23 -0600 -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Status: OR - -On Mon, 2003-01-06 at 05:36, Shridhar Daithankar wrote: -> On 6 Jan 2003 at 12:22, Ulrich Neumann wrote: -> -> > Hello all, -> > If someone is interested in the code I can send a zip file to everyone -> > who wants. -> -> I suggest you preserver your work. The reason I suggested thread are mainly two -> folds. -> -> 1) Get I/O time used fuitfully - - -AIO may address this without the need for integrated threading. -Arguably, from the long thread that last appeared on the topic of AIO, -some hold that AIO doesn't even offer anything beyond the current -implementation. As such, it's highly doubtful that integrated threading -is going to offer anything beyond what a sound AIO implementation can -achieve. - - -> 2) Use multiple CPU better. -> - - -Multiple processes tend to universally support multiple CPUs better than -does threading. On some platforms, the level of threading support is -currently only user mode implementations which means no additional CPU -use. Furthermore, some platforms where user-mode threads are defacto, -they don't even allow for scheduling bias resulting is less work being -accomplished within the same time interval (work slice must be divided -between n-threads within the process, all of which run on a single CPU). - - -> It will not require as much code cleaning as your efforts might had. However -> your work will be very useful if somebody decides to use thread in any fashion -> in core postgresql. -> -> I was hoping for bit more optimistic response given that what I suggested was -> totally optional at any point of time but very important from performance -> point. Besides the change would have been gradual as required.. -> - - -Speaking for my self, I probably would of been more excited if the -offered framework had addressed several issues. The short list is: - -o Code needs to be more robust. It shouldn't be calling exit directly -as, I believe, it should be allowing for PostgreSQL to clean up some. -Correct me as needed. I would of also expected the code of adopted -PostgreSQL's semantics and mechanisms as needed (error reporting, etc). -I do understand it was an initial attempt to simply get something in -front of some eyes and have something to talk about. Just the same, I -was expecting something that we could actually pull the trigger with. - -o Code isn't very portable. Looked fairly okay for pthread platforms, -however, there is new emphasis on the Win32 platform. I think it would -be a mistake to introduce something as significant as threading without -addressing Win32 from the get-go. - -o I would desire a more highly abstracted/portable interface which -allows for different threading and synchronization primitives to be -used. Current implementation is tightly coupled to pthreads. -Furthermore, on platforms such as Solaris, I would hope it would easily -allow for plugging in its native threading primitives which are touted -to be much more efficient than pthreads on said platform. - -o Code is not commented. I would hope that adding new code for -something as important as threading would be commented. - -o Code is fairly trivial and does not address other primitives -(semaphores, mutexs, conditions, TSS, etc) portably which would be -required for anything but the most trivial of threaded work. This is -especially true in such an application where data IS the application. -As such, you must reasonably assume that threads need some form of -portable serialization primitives, not to mention mechanisms for -non-trivial communication. - -o Does not address issues such as thread signaling or status reporting. - -o Pool interface is rather simplistic. Does not currently support -concepts such as wake pool, stop pool, pool status, assigning a pool to -work, etc. In fact, it's not altogether obvious what the capabilities -intent is of the current pool implementation. - -o Doesn't seem to address any form of thread communication facilities -(mailboxes, queues, etc). - - -There are probably other things that I can find if I spend more than -just a couple of minutes looking at the code. Honestly, I love threads -but I can see that the current code offering is not much more than a -token in its current form. No offense meant. - -After it's all said and done, I'd have to see a lot more meat before I'd -be convinced that threading is ready for PostgreSQL; from both a social -and technological perspective. - - -Regards, - --- -Greg Copeland -Copeland Computer Consulting - - ----------------------------(end of broadcast)--------------------------- -TIP 2: you can get off all lists at once with the unregister command - (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) - -From pgsql-hackers-owner+M33899@postgresql.org Tue Jan 7 03:00:25 2003 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h0780Mi00624 - for ; Tue, 7 Jan 2003 03:00:23 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id 46FA747687C; Tue, 7 Jan 2003 03:00:21 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 69717475F25; Tue, 7 Jan 2003 03:00:13 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id 5323E475F39 - for ; Tue, 7 Jan 2003 03:00:01 -0500 (EST) -Received: from www.pspl.co.in (www.pspl.co.in [202.54.11.65]) - by postgresql.org (Postfix) with ESMTP id 351DD475EE1 - for ; Tue, 7 Jan 2003 02:59:58 -0500 (EST) -Received: (from root@localhost) - by www.pspl.co.in (8.11.6/8.11.6) id h077xvs03265 - for ; Tue, 7 Jan 2003 13:29:57 +0530 -Received: from daithan (daithan.intranet.pspl.co.in [192.168.7.161]) - by www.pspl.co.in (8.11.6/8.11.0) with ESMTP id h077xvr03260 - for ; Tue, 7 Jan 2003 13:29:57 +0530 -From: "Shridhar Daithankar" -To: " -Date: Tue, 07 Jan 2003 13:30:05 +0530 -MIME-Version: 1.0 -Subject: Re: [HACKERS] Threads -Reply-To: shridhar_daithankar@persistent.co.in -Message-ID: <3E1AD65D.10112.192793@localhost> -References: <3E19B78B.25689.15BFFE@localhost> -In-Reply-To: <1041857302.17321.49.camel@mouse.copelandconsulting.net> -X-Mailer: Pegasus Mail for Windows (v4.02) -Content-Type: text/plain; charset=US-ASCII -Content-Transfer-Encoding: 7BIT -Content-Description: Mail message body -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Status: OR - -On 6 Jan 2003 at 6:48, Greg Copeland wrote: -> > 1) Get I/O time used fuitfully -> AIO may address this without the need for integrated threading. -> Arguably, from the long thread that last appeared on the topic of AIO, -> some hold that AIO doesn't even offer anything beyond the current -> implementation. As such, it's highly doubtful that integrated threading -> is going to offer anything beyond what a sound AIO implementation can -> achieve. - -Either way, a complete aio or threading implementation is not available on -major platforms that postgresql runs. Linux definitely does not have one, last -I checked. - -If postgresql is not using aio or threading, we should start using one of them, -is what I feel. What do you say? - -> > 2) Use multiple CPU better. -> Multiple processes tend to universally support multiple CPUs better than -> does threading. On some platforms, the level of threading support is -> currently only user mode implementations which means no additional CPU -> use. Furthermore, some platforms where user-mode threads are defacto, -> they don't even allow for scheduling bias resulting is less work being -> accomplished within the same time interval (work slice must be divided -> between n-threads within the process, all of which run on a single CPU). - -The frame-work I have posted, threading is optional at build and should be a -configuration option if it gets integrated. So for the platforms that can not -spread threads across multiple CPUs, it can simply be turned off.. - -> Speaking for my self, I probably would of been more excited if the -> offered framework had addressed several issues. The short list is: -> -> o Code needs to be more robust. It shouldn't be calling exit directly -> as, I believe, it should be allowing for PostgreSQL to clean up some. -> Correct me as needed. I would of also expected the code of adopted -> PostgreSQL's semantics and mechanisms as needed (error reporting, etc). -> I do understand it was an initial attempt to simply get something in -> front of some eyes and have something to talk about. Just the same, I -> was expecting something that we could actually pull the trigger with. - -That could be done. - -> -> o Code isn't very portable. Looked fairly okay for pthread platforms, -> however, there is new emphasis on the Win32 platform. I think it would -> be a mistake to introduce something as significant as threading without -> addressing Win32 from the get-go. - -If you search for "pthread" in thread.c, there are not many instances. Same -goes for thread.h. From what I understand windows threading, it would be less -than 10 minutes job to #ifdef the pthread related part on either file. - -It is just that I have not played with windows threading and nor I am inclined -to...;-) - -> -> o I would desire a more highly abstracted/portable interface which -> allows for different threading and synchronization primitives to be -> used. Current implementation is tightly coupled to pthreads. -> Furthermore, on platforms such as Solaris, I would hope it would easily -> allow for plugging in its native threading primitives which are touted -> to be much more efficient than pthreads on said platform. - -Same as above. If there can be two cases separated with #ifdef, there can be -more.. But what is important is to have a thread that can be woken up as and -when required with any function desired. That is the basic idea. - -> o Code is not commented. I would hope that adding new code for -> something as important as threading would be commented. - -Agreed. - -> o Code is fairly trivial and does not address other primitives -> (semaphores, mutexs, conditions, TSS, etc) portably which would be -> required for anything but the most trivial of threaded work. This is -> especially true in such an application where data IS the application. -> As such, you must reasonably assume that threads need some form of -> portable serialization primitives, not to mention mechanisms for -> non-trivial communication. - -I don't get this. Probably I should post a working example. It is not threads -responsibility to make a function thread safe which is changed on the fly. The -function has to make sure that it is thread safe. That is altogether different -effort.. - -> o Does not address issues such as thread signaling or status reporting. - ->From what I learnt from pthreads on linux, I would not mix threads and signals. -One can easily add code in runner function that disables any signals for thread -while the thread starts running. This would leave original signal handling -mechanism in place. - -As far as status reporting is concerned, the thread sould be initiated while -back-end starts and terminated with backend termination. What is about status -reporting? - -> o Pool interface is rather simplistic. Does not currently support -> concepts such as wake pool, stop pool, pool status, assigning a pool to -> work, etc. In fact, it's not altogether obvious what the capabilities -> intent is of the current pool implementation. - -Could you please elaborate? I am using same interface in c++ for a server -application and never faced a problem like that..;-) - - -> o Doesn't seem to address any form of thread communication facilities -> (mailboxes, queues, etc). - -Not part of this abstraction of threading mechanism. Intentionally left out to -keep things clean. - -> There are probably other things that I can find if I spend more than -> just a couple of minutes looking at the code. Honestly, I love threads -> but I can see that the current code offering is not much more than a -> token in its current form. No offense meant. - -None taken. Point is it is useful and that is enough for me. If you could -elaborate examples for any problems you see, I can probably modify it. (Code -documentation is what I will do now) - -> After it's all said and done, I'd have to see a lot more meat before I'd -> be convinced that threading is ready for PostgreSQL; from both a social -> and technological perspective. - -Tell me about it.. - - -Bye - Shridhar - --- -What's this script do? unzip ; touch ; finger ; mount ; gasp ; yes ; umount -; sleepHint for the answer: not everything is computer-oriented. Sometimes -you'rein a sleeping bag, camping out.(Contributed by Frans van der Zande.) - - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M33921@postgresql.org Tue Jan 7 11:10:53 2003 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h07GApX13277 - for ; Tue, 7 Jan 2003 11:10:51 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id 5BDC0477200; Tue, 7 Jan 2003 11:06:58 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 9EE41477268; Tue, 7 Jan 2003 11:06:40 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id ACEA5477260 - for ; Tue, 7 Jan 2003 11:06:35 -0500 (EST) -Received: from CopelandConsulting.Net (dsl-24293-ld.customer.centurytel.net [209.142.135.135]) - by postgresql.org (Postfix) with ESMTP id 78B51477165 - for ; Tue, 7 Jan 2003 11:06:28 -0500 (EST) -Received: from [192.168.1.2] (mouse.copelandconsulting.net [192.168.1.2]) - by CopelandConsulting.Net (8.10.1/8.10.1) with ESMTP id h07G68711510; - Tue, 7 Jan 2003 10:06:09 -0600 (CST) -X-Trade-Id: -To: shridhar_daithankar@persistent.co.in -cc: " -In-Reply-To: <3E1AD65D.10112.192793@localhost> -References: <3E19B78B.25689.15BFFE@localhost> - <3E1AD65D.10112.192793@localhost> -Content-Type: text/plain -Organization: Copeland Computer Consulting -Message-ID: <1041955572.17639.148.camel@mouse.copelandconsulting.net> -MIME-Version: 1.0 -X-Mailer: Ximian Evolution 1.2.0 -Date: 07 Jan 2003 10:06:12 -0600 -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Status: OR - -On Tue, 2003-01-07 at 02:00, Shridhar Daithankar wrote: -> On 6 Jan 2003 at 6:48, Greg Copeland wrote: -> > > 1) Get I/O time used fuitfully -> > AIO may address this without the need for integrated threading. -> > Arguably, from the long thread that last appeared on the topic of AIO, -> > some hold that AIO doesn't even offer anything beyond the current -> > implementation. As such, it's highly doubtful that integrated threading -> > is going to offer anything beyond what a sound AIO implementation can -> > achieve. -> -> Either way, a complete aio or threading implementation is not available on -> major platforms that postgresql runs. Linux definitely does not have one, last -> I checked. -> - -There are two or three significant AIO implementation efforts currently -underway for Linux. One such implementation is available from the Red -Hat Server Edition (IIRC) and has been available for some time now. I -believe Oracle is using it. SGI also has an effort and I forget where -the other one comes from. Nonetheless, I believe it's going to be a -hard fought battle to get AIO implemented simply because I don't think -anyone, yet, can truly argue a case on the gain vs effort. - -> If postgresql is not using aio or threading, we should start using one of them, -> is what I feel. What do you say? -> - -I did originally say that I'd like to see an AIO implementation. Then -again, I don't current have a position to stand other than simply saying -it *might* perform better. ;) Not exactly a position that's going to -win the masses over. - -> > was expecting something that we could actually pull the trigger with. -> -> That could be done. -> - -I'm sure it can, but that's probably the easiest item to address. - -> > -> > o Code isn't very portable. Looked fairly okay for pthread platforms, -> > however, there is new emphasis on the Win32 platform. I think it would -> > be a mistake to introduce something as significant as threading without -> > addressing Win32 from the get-go. -> -> If you search for "pthread" in thread.c, there are not many instances. Same -> goes for thread.h. From what I understand windows threading, it would be less -> than 10 minutes job to #ifdef the pthread related part on either file. -> -> It is just that I have not played with windows threading and nor I am inclined -> to...;-) -> - -Well, the method above is going to create a semi-ugly mess. I've -written thread abstraction layers which cover OS/2, NT, and pthreads. -Each have subtle distinction. What really needs to be done is the -creation of another abstraction layer which your current code would sit -on top of. That way, everything contained within is clear and easy to -read. The big bonus is that as additional threading implementations -need to be added, only the "low-level" abstraction stuff needs to -modified. Done properly, each thread implementation would be it's own -module requiring little #if clutter. - -As you can see, that's a fair amount of work and far from where the code -currently is. - -> > -> > o I would desire a more highly abstracted/portable interface which -> > allows for different threading and synchronization primitives to be -> > used. Current implementation is tightly coupled to pthreads. -> > Furthermore, on platforms such as Solaris, I would hope it would easily -> > allow for plugging in its native threading primitives which are touted -> > to be much more efficient than pthreads on said platform. -> -> Same as above. If there can be two cases separated with #ifdef, there can be -> more.. But what is important is to have a thread that can be woken up as and -> when required with any function desired. That is the basic idea. -> - -Again, there's a lot of work in creating a well formed abstraction layer -for all of the mechanics that are required. Furthermore, different -thread implementations have slightly different semantics which further -complicates things. Worse, some types of primitives are simply not -available with some thread implementations. That means those platforms -require it to be written from the primitives that are available on the -platform. Yet more work. - - -> > o Code is fairly trivial and does not address other primitives -> > (semaphores, mutexs, conditions, TSS, etc) portably which would be -> > required for anything but the most trivial of threaded work. This is -> > especially true in such an application where data IS the application. -> > As such, you must reasonably assume that threads need some form of -> > portable serialization primitives, not to mention mechanisms for -> > non-trivial communication. -> -> I don't get this. Probably I should post a working example. It is not threads -> responsibility to make a function thread safe which is changed on the fly. The -> function has to make sure that it is thread safe. That is altogether different -> effort.. - - -You're right, it's not the thread's responsibility, however, it is the -threading toolkit's. In this case, you're offering to be the toolkit -which functions across two platforms, just for starters. Reasonably, -you should expect a third to quickly follow. - -> -> > o Does not address issues such as thread signaling or status reporting. -> -> >From what I learnt from pthreads on linux, I would not mix threads and signals. -> One can easily add code in runner function that disables any signals for thread -> while the thread starts running. This would leave original signal handling -> mechanism in place. -> -> As far as status reporting is concerned, the thread sould be initiated while -> back-end starts and terminated with backend termination. What is about status -> reporting? -> -> > o Pool interface is rather simplistic. Does not currently support -> > concepts such as wake pool, stop pool, pool status, assigning a pool to -> > work, etc. In fact, it's not altogether obvious what the capabilities -> > intent is of the current pool implementation. -> -> Could you please elaborate? I am using same interface in c++ for a server -> application and never faced a problem like that..;-) -> -> -> > o Doesn't seem to address any form of thread communication facilities -> > (mailboxes, queues, etc). -> -> Not part of this abstraction of threading mechanism. Intentionally left out to -> keep things clean. -> -> > There are probably other things that I can find if I spend more than -> > just a couple of minutes looking at the code. Honestly, I love threads -> > but I can see that the current code offering is not much more than a -> > token in its current form. No offense meant. -> -> None taken. Point is it is useful and that is enough for me. If you could -> elaborate examples for any problems you see, I can probably modify it. (Code -> documentation is what I will do now) -> -> > After it's all said and done, I'd have to see a lot more meat before I'd -> > be convinced that threading is ready for PostgreSQL; from both a social -> > and technological perspective. -> -> Tell me about it.. -> - -Long story short, if PostgreSQL is to use threads, it shouldn't be -handicapped by having a very limited subset of functionality. With the -code that has been currently submitted, I don't believe you could even -effectively implement a parallel sort. - -To get an idea of the types of things that would be needed, check out -the ACE Toolkit. There are a couple of other fairly popular toolkits as -well. Nonetheless, it's a significant effort and the current code is a -long ways off from being usable. - - --- -Greg Copeland -Copeland Computer Consulting - - ----------------------------(end of broadcast)--------------------------- -TIP 5: Have you checked our extensive FAQ? - -http://www.postgresql.org/users-lounge/docs/faq.html - -From pgsql-hackers-owner+M33944@postgresql.org Tue Jan 7 13:22:04 2003 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h07IM2X05350 - for ; Tue, 7 Jan 2003 13:22:02 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id 544EF476AC1; Tue, 7 Jan 2003 13:22:05 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 341134761E8; Tue, 7 Jan 2003 13:21:55 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id 48974475ADE - for ; Tue, 7 Jan 2003 13:21:50 -0500 (EST) -Received: from sabre.velocet.net (sabre.velocet.net [216.138.209.205]) - by postgresql.org (Postfix) with ESMTP id B8D40475AD7 - for ; Tue, 7 Jan 2003 13:21:49 -0500 (EST) -Received: from stark.dyndns.tv (H162.C233.tor.velocet.net [216.138.233.162]) - by sabre.velocet.net (Postfix) with ESMTP - id 887681382B9; Tue, 7 Jan 2003 13:21:48 -0500 (EST) -Received: from localhost - ([127.0.0.1] helo=stark.dyndns.tv ident=foobar) - by stark.dyndns.tv with smtp (Exim 3.36 #1 (Debian)) - id 18VyMF-0002zN-00; Tue, 07 Jan 2003 13:21:47 -0500 -To: Greg Copeland -cc: kar@kakidata.dk, PGHackers -Subject: Re: [HACKERS] Threads -References: - <23763.1041644042@sss.pgh.pa.us> - <1041646276.15927.202.camel@mouse.copelandconsulting.net> - <200301041359.35715.kar@kakidata.dk> - <1041704480.15927.224.camel@mouse.copelandconsulting.net> -In-Reply-To: <1041704480.15927.224.camel@mouse.copelandconsulting.net> -From: Greg Stark -Organization: The Emacs Conspiracy; member since 1992 -Date: 07 Jan 2003 13:21:47 -0500 -Message-ID: <87isx0izwk.fsf@stark.dyndns.tv> -Lines: 43 -User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.2 -MIME-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Status: OR - - -Greg Copeland writes: - -> That's the power of using the process model that is currently in use. Should -> it do something naughty, we bitch and complain politely, throw our hands in -> the air and exit. We no longer have to worry about the state and validity of -> that backend. - -You missed the point of his post. If one process in your database does -something nasty you damn well should worry about the state of and validity of -the entire database, not just that one backend. - -Are you really sure you caught the problem before it screwed up the data in -shared memory? On disk? - - -This whole topic is in need of some serious FUD-dispelling and careful -analysis. Here's a more calm explanation of the situation on this particular -point. Perhaps I'll follow up with something on IO concurrency later. - -The point in consideration here is really memory isolation. Threads by default -have zero isolation between threads. They can all access each other's memory -even including their stack. Most of that memory is in fact only needed by a -single thread. - -Processes by default have complete memory isolation. However postgres actually -weakens that by doing a lot of work in a shared memory pool. That memory gets -exactly the same protection as it would get in a threaded model, which is to -say none. - -So the reality is that if you have a bug most likely you've only corrupted the -local data which can be easily cleaned up either way. In the thread model -there's also the unlikely but scary risk that you've damaged other threads' -memory. And in either case there's the possibility that you've damaged the -shared pool which is unrecoverable. - -In theory minimising the one case of corrupting other threads' local data -shouldn't make a big difference to the risk in the case of an assertion -failure. I'm not sure in practice if that's true though. Processes probably -reduce the temptation to do work in the shared area too. - --- -greg - - ----------------------------(end of broadcast)--------------------------- -TIP 2: you can get off all lists at once with the unregister command - (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) - -From pgsql-hackers-owner+M33945@postgresql.org Tue Jan 7 13:48:12 2003 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h07Im8X15155 - for ; Tue, 7 Jan 2003 13:48:08 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id BF0454773D3; Tue, 7 Jan 2003 13:43:10 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 781634773A6; Tue, 7 Jan 2003 13:43:03 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id 28074477390 - for ; Tue, 7 Jan 2003 13:42:59 -0500 (EST) -Received: from CopelandConsulting.Net (dsl-24293-ld.customer.centurytel.net [209.142.135.135]) - by postgresql.org (Postfix) with ESMTP id 1392B476682 - for ; Tue, 7 Jan 2003 13:42:42 -0500 (EST) -Received: from [192.168.1.2] (mouse.copelandconsulting.net [192.168.1.2]) - by CopelandConsulting.Net (8.10.1/8.10.1) with ESMTP id h07IgS715128; - Tue, 7 Jan 2003 12:42:28 -0600 (CST) -X-Trade-Id: -To: Greg Stark -cc: kar@kakidata.dk, PGHackers -In-Reply-To: <87isx0izwk.fsf@stark.dyndns.tv> -References: - <23763.1041644042@sss.pgh.pa.us> - <1041646276.15927.202.camel@mouse.copelandconsulting.net> - <200301041359.35715.kar@kakidata.dk> - <1041704480.15927.224.camel@mouse.copelandconsulting.net> - <87isx0izwk.fsf@stark.dyndns.tv> -Content-Type: text/plain -Organization: Copeland Computer Consulting -Message-ID: <1041964952.29180.10.camel@mouse.copelandconsulting.net> -MIME-Version: 1.0 -X-Mailer: Ximian Evolution 1.2.0 -Date: 07 Jan 2003 12:42:33 -0600 -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Status: OR - -On Tue, 2003-01-07 at 12:21, Greg Stark wrote: -> Greg Copeland writes: -> -> > That's the power of using the process model that is currently in use. Should -> > it do something naughty, we bitch and complain politely, throw our hands in -> > the air and exit. We no longer have to worry about the state and validity of -> > that backend. -> -> You missed the point of his post. If one process in your database does -> something nasty you damn well should worry about the state of and validity of -> the entire database, not just that one backend. -> - -I can assure you I did not miss the point. No idea why you're -continuing to spell it out. In this case, it appears the quotation is -being taken out of context or it was originally stated in an improper -context. - -> Are you really sure you caught the problem before it screwed up the data in -> shared memory? On disk? -> -> -> This whole topic is in need of some serious FUD-dispelling and careful -> analysis. Here's a more calm explanation of the situation on this particular -> point. Perhaps I'll follow up with something on IO concurrency later. -> - - -Hmmm. Not sure what needs to be dispelled since I've not seen any FUD. - - -> The point in consideration here is really memory isolation. Threads by default -> have zero isolation between threads. They can all access each other's memory -> even including their stack. Most of that memory is in fact only needed by a -> single thread. -> - -Again, this has been covered already. - - -> Processes by default have complete memory isolation. However postgres actually -> weakens that by doing a lot of work in a shared memory pool. That memory gets -> exactly the same protection as it would get in a threaded model, which is to -> say none. -> - -Again, this has all been covered, more or less. You're comments seem to -imply that you did not fully read what has been said on the topic thus -far or that you misunderstood something that was said. Of course, it's -also possible that I may of said something out of it's proper context -which may be confusing you. - -I think it's safe to say I don't have any further comment unless -something new is being brought to the table. Should there be something -new to cover, I'm happy to talk about it. At this point, however, it -appears that it's been beat to death already. - - --- -Greg Copeland -Copeland Computer Consulting - - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate -subscribe-nomail command to majordomo@postgresql.org so that your -message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M33946@postgresql.org Tue Jan 7 14:02:33 2003 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h07J2TX22478 - for ; Tue, 7 Jan 2003 14:02:30 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id 6A905477204; Tue, 7 Jan 2003 14:02:32 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 3546E476688; Tue, 7 Jan 2003 14:02:21 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id E3CC44760BD - for ; Tue, 7 Jan 2003 14:02:14 -0500 (EST) -Received: from sss.pgh.pa.us (unknown [192.204.191.242]) - by postgresql.org (Postfix) with ESMTP id 3D8FA475AD7 - for ; Tue, 7 Jan 2003 14:02:14 -0500 (EST) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.12.6/8.12.6) with ESMTP id h07J1s0U019750; - Tue, 7 Jan 2003 14:01:54 -0500 (EST) -To: Greg Stark -cc: Greg Copeland , kar@kakidata.dk, - PGHackers -Subject: Re: [HACKERS] Threads -In-Reply-To: <87isx0izwk.fsf@stark.dyndns.tv> -References: <23763.1041644042@sss.pgh.pa.us> <1041646276.15927.202.camel@mouse.copelandconsulting.net> <200301041359.35715.kar@kakidata.dk> <1041704480.15927.224.camel@mouse.copelandconsulting.net> <87isx0izwk.fsf@stark.dyndns.tv> -Comments: In-reply-to Greg Stark - message dated "07 Jan 2003 13:21:47 -0500" -Date: Tue, 07 Jan 2003 14:01:53 -0500 -Message-ID: <19749.1041966113@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Status: OR - -Greg Stark writes: -> You missed the point of his post. If one process in your database does -> something nasty you damn well should worry about the state of and validity of -> the entire database, not just that one backend. - -Right. And in fact we do blow away all the processes when any one of -them crashes or panics. Nonetheless, memory isolation between processes -is a Good Thing, because it reduces the chances that a process gone -wrong will cause damage via other processes before they can be shut -down. - -Here is a simple example of a scenario where that isolation buys us -something: suppose that we have a bug that tromps on memory starting at -some point X until it falls off the sbrk boundary and dumps core. -(There are plenty of ways to make that happen, such as miscalculating -the length of a memcpy or memset operation as -1.) Such a bug causes -no serious damage in isolation, because the process suffering the -failure will be in a tight data-copying or data-zeroing loop until it -gets the SIGSEGV exception. It won't do anything bad based on all the -data structures it has clobbered during its march to the end of memory. - -However, put that same bug in a multithreading context, and it becomes -entirely possible that some other thread will be dispatched and will -try to make use of already-clobbered data structures before the ultimate -SIGSEGV exception happens. Now you have the potential for unlimited -trouble. - -In general, isolation buys you some safety anytime there is a delay -between the occurrence of a failure and its detection. - -> Processes by default have complete memory isolation. However postgres -> actually weakens that by doing a lot of work in a shared memory -> pool. That memory gets exactly the same protection as it would get in -> a threaded model, which is to say none. - -Yes. We try to minimize the risk by keeping the shared memory pool -relatively small and not doing more than we have to in it. (For -example, this was one of the arguments against creating a shared plan -cache.) It's also very helpful that in most platforms, shared memory -is not address-wise contiguous to normal memory; thus for example a -process caught in a memset death march will hit a SIGSEGV before it -gets to the shared memory block. - -It's interesting to note that this can be made into an argument for -not making shared_buffers very large: the larger the fraction of your -address space that the shared buffers occupy, the larger the chance -that a wild store will overwrite something you'd wish it didn't. -I can't recall anyone having made that point during our many discussions -of appropriate shared_buffer sizing. - -> So the reality is that if you have a bug most likely you've only corrupted the -> local data which can be easily cleaned up either way. In the thread model -> there's also the unlikely but scary risk that you've damaged other threads' -> memory. And in either case there's the possibility that you've damaged the -> shared pool which is unrecoverable. - -In a thread model, *most* of the accessible memory space would be shared -with other threads, at least potentially. So I think you're wrong to -categorize the second case as unlikely. - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 6: Have you searched our list archives? - -http://archives.postgresql.org - -From pgsql-hackers-owner+M37860@postgresql.org Fri Apr 11 15:37:03 2003 -Return-path: -Received: from relay3.pgsql.com (relay3.pgsql.com [64.117.224.149]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h3BJaxv13018 - for ; Fri, 11 Apr 2003 15:37:01 -0400 (EDT) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by relay3.pgsql.com (Postfix) with ESMTP - id 3F9D0EA81E7; Fri, 11 Apr 2003 19:36:56 +0000 (GMT) -X-Original-To: pgsql-hackers@postgresql.org -Received: from spampd.localdomain (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id D27B2476036 - for ; Fri, 11 Apr 2003 15:35:32 -0400 (EDT) -Received: from mail1.ihs.com (mail1.ihs.com [170.207.70.222]) - by postgresql.org (Postfix) with ESMTP id 742DD475F5F - for ; Fri, 11 Apr 2003 15:35:31 -0400 (EDT) -Received: from css120.ihs.com (css120.ihs.com [170.207.105.120]) - by mail1.ihs.com (8.12.9/8.12.9) with ESMTP id h3BJZHRF027332; - Fri, 11 Apr 2003 13:35:17 -0600 (MDT) -Date: Fri, 11 Apr 2003 13:31:06 -0600 (MDT) -From: "scott.marlowe" -To: Ron Peacetree -cc: -Subject: Re: [HACKERS] Anyone working on better transaction locking? -In-Reply-To: -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -X-MailScanner: Found to be clean -X-Spam-Status: No, hits=-31.5 required=5.0 - tests=BAYES_10,EMAIL_ATTRIBUTION,IN_REP_TO,QUOTED_EMAIL_TEXT, - QUOTE_TWICE_1,REPLY_WITH_QUOTES,USER_AGENT_PINE - autolearn=ham version=2.50 -X-Spam-Checker-Version: SpamAssassin 2.50 (1.173-2003-02-20-exp) -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -On Wed, 9 Apr 2003, Ron Peacetree wrote: - -> "Andrew Sullivan" wrote in message -> news:20030409170926.GH2255@libertyrms.info... -> > On Wed, Apr 09, 2003 at 05:41:06AM +0000, Ron Peacetree wrote: -> > Nonsense. You explicitly made the MVCC comparison with Oracle, and -> > are asking for a "better" locking mechanism without providing any -> > evidence that PostgreSQL's is bad. -> > -> Just because someone else's is "better" does not mean PostgreSQL's is -> "bad", and I've never said such. As I've said, I'll get back to Tom -> and the list on this. - -But you didn't identify HOW it was better. I think that's the point -being made. - -> > > Please see my posts with regards to ... -> > -> > I think your other posts were similar to the one which started this -> > thread: full of mighty big pronouncements which turned out to depend -> > on a bunch of not-so-tenable assumptions. -> > -> Hmmm. Well, I don't think of algorithm analysis by the likes of -> Knuth, Sedgewick, Gonnet, and Baeza-Yates as being "not so tenable -> assumptions", but YMMV. As for "mighty pronouncements", that also -> seems a bit misleading since we are talking about quantifiable -> programming and computer science issues, not unquantifiable things -> like politics. - -But the real truth is revealed when the rubber hits the pavement. -Remember that Linux Torvalds was roundly criticized for his choice of a -monolithic development model for his kernel, and was literally told that -his choice would restrict to "toy" status and that no commercial OS could -scale with a monolithic kernel. - -There's no shortage of people with good ideas, just people with the skills -to implement those good ideas. If you've got a patch to apply that's been -tested to show something is faster EVERYONE here wants to see it. - -If you've got a theory, no matter how well backed up by academic research, -it's still just a theory. Until someone writes to code to implement it, -the gains are theoretical, and many things that MIGHT help don't because -of the real world issues underlying your database, like I/O bandwidth or -CPU <-> memory bandwidth. - -> > I'm sorry to be so cranky about this, but I get tired of having to -> > defend one of my employer's core technologies from accusations based -> > on half-truths and "everybody knows" assumptions. For instance, -> > -> Again, "accusations" is a bit strong. I thought the discussion was -> about the technical merits and costs of various features and various -> ways to implement them, particularly when this product must compete -> for installed base with other solutions. Being coldly realistic about -> what a product's strengths and weaknesses are is, again, just good -> business. Sun Tzu's comment about knowing the enemy and yourself -> seems appropriate here... - -No, you're wrong. Postgresql doesn't have to compete. It doesn't have to -win. it doesn't need a marketing department. All those things are nice, -and I'm glad if it does them, but doesn't HAVE TO. Postgresql has to -work. It does that well. - -Postgresql CAN compete if someone wants to put the effort into competing, -but it isn't a priority for me. Working is the priority, and if other -people aren't smart enough to test Postgresql to see if it works for them, -all the better, I keep my edge by having a near zero cost database engine, -while the competition spends money on MSSQL or Oracle. - -Tom and Andrew ARE coldly realistic about the shortcomings of postgresql. -It has issues, and things that need to be fixed. It needs more coders. -It doesn't need every feature that Oracle or DB2 have. Heck some of their -"features" would be considered a mis-feature in the Postgresql world. - -> > > I'll mention thread support in passing, -> > -> > there's actually a FAQ item about thread support, because in the -> > opinion of those who have looked at it, the cost is just not worth -> > the benefit. If you have evidence to the contrary (specific -> > evidence, please, for this application), and have already read all -> the -> > previous discussion of the topic, perhaps people would be interested -> in -> > opening that debate again (though I have my doubts). -> > -> Zeus had a performance ceiling roughly 3x that of Apache when Zeus -> supported threading as well as pre-forking and Apache only supported -> pre forking. The Apache folks now support both. DB2, Oracle, and SQL -> Server all use threads. Etc, etc. - -Yes, and if you configured your apache server to have 20 or 30 spare -servers, in the real world, it was nearly neck and neck to Zeus, but since -Zeus cost like $3,000 a copy, it is still cheaper to just overwhelm it -with more servers running apache than to use zeus. - -> That's an awful lot of very bright programmers and some serious $$ -> voting that threads are worth it. - -For THAT application. for what a web server does, threads can be very -useful, even useful enough to put up with the problems created by running -threads on multiple threading libs on different OSes. - -Let me ask you, if Zeus scrams and crashes out, and it's installed -properly so it just comes right back up, how much data can you lose? - -If Postgresql scrams and crashes out, how much data can you lost? - -> Given all that, if PostgreSQL -> specific -> thread support is =not= showing itself to be a win that's an -> unexpected -> enough outcome that we should be asking hard questions as to why not. - -There HAS been testing on threads in Postgresql. It has been covered to -death. The fact that you're still arguing proves you likely haven't read -the archive (google has it back to way back when, use that to look it up) -about this subject. - -Threads COULD help on multi-sorted results, and a few other areas, but the -increase in performance really wasn't that great for 95% of all the cases, -and for the 5% it was, simple query planner improvements have provided far -greater performance increases. - -The problem with threading is that we can either use the one process -> -many thread design, which I personally don't trust for something like a -database, or a process per backend connection which can run -multi-threaded. This scenario makes Postgresql just as stable and -reliable as it was as a multi-process app, but allows threaded performance -in certain areas of the backend that are parallelizable to run in parallel -on multi-CPU systems. - -the gain, again, is minimal, and on a system with many users accessing it, -there is NO real world gain. - -> At their core, threads are a context switching efficiency tweak. - -Except that on the two OSes which Postgresql runs on the most, threads are -really no faster than processes. In the Linux kernel, the only real -difference is how the OS treats them, creation, destruction of threads -versus processes is virtually identical there. - -> Certainly it's =possible= that threads have nothing to offer -> PostgreSQL, but IMHO it's not =probable=. Just another thing for me -> to add to my TODO heap for looking at... - -It's been tested, it didn't help a lot, and it made it MUCH harder to -maintain, as threads in Linux are handled by a different lib than in say -Solaris, or Windows or any other OS. I.e. you can't guarantee the thread -lib you need will be there, and that there are no bugs. MySQL still has -thread bug issues pop up, most of which are in the thread libs themselves. - - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M37865@postgresql.org Fri Apr 11 17:34:21 2003 -Return-path: -Received: from relay1.pgsql.com (relay1.pgsql.com [64.49.215.129]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h3BLYIv28485 - for ; Fri, 11 Apr 2003 17:34:19 -0400 (EDT) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by relay1.pgsql.com (Postfix) with ESMTP - id 0AF036F77ED; Fri, 11 Apr 2003 17:34:19 -0400 (EDT) -X-Original-To: pgsql-hackers@postgresql.org -Received: from spampd.localdomain (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id EBB41476323 - for ; Fri, 11 Apr 2003 17:33:02 -0400 (EDT) -Received: from filer (12-234-86-219.client.attbi.com [12.234.86.219]) - by postgresql.org (Postfix) with ESMTP id CED7D4762E1 - for ; Fri, 11 Apr 2003 17:32:57 -0400 (EDT) -Received: from localhost (localhost [127.0.0.1]) - (uid 1000) - by filer with local; Fri, 11 Apr 2003 14:32:59 -0700 -Date: Fri, 11 Apr 2003 14:32:59 -0700 -From: Kevin Brown -To: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Anyone working on better transaction locking? -Message-ID: <20030411213259.GU1833@filer> -Mail-Followup-To: Kevin Brown , - pgsql-hackers@postgresql.org -References: <20030409170926.GH2255@libertyrms.info> -MIME-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -Content-Transfer-Encoding: 7bit -Content-Disposition: inline -In-Reply-To: -User-Agent: Mutt/1.4i -Organization: Frobozzco International -X-Spam-Status: No, hits=-38.0 required=5.0 - tests=BAYES_10,EMAIL_ATTRIBUTION,IN_REP_TO,QUOTED_EMAIL_TEXT, - REFERENCES,REPLY_WITH_QUOTES,USER_AGENT_MUTT - autolearn=ham version=2.50 -X-Spam-Checker-Version: SpamAssassin 2.50 (1.173-2003-02-20-exp) -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -Ron Peacetree wrote: -> Zeus had a performance ceiling roughly 3x that of Apache when Zeus -> supported threading as well as pre-forking and Apache only supported -> pre forking. The Apache folks now support both. DB2, Oracle, and SQL -> Server all use threads. Etc, etc. - -You can't use Apache as an example of why you should thread a database -engine, except for the cases where the database is used much like the -web server is: for numerous short transactions. - -> That's an awful lot of very bright programmers and some serious $$ -> voting that threads are worth it. Given all that, if PostgreSQL -> specific thread support is =not= showing itself to be a win that's -> an unexpected enough outcome that we should be asking hard questions -> as to why not. - -It's not that there won't be any performance benefits to be had from -threading (there surely will, on some platforms), but gaining those -benefits comes at a very high development and maintenance cost. You -lose a *lot* of robustness when all of your threads share the same -memory space, and make yourself vulnerable to classes of failures that -simply don't happen when you don't have shared memory space. - -PostgreSQL is a compromise in this regard: it *does* share memory, but -it only shares memory that has to be shared, and nothing else. To get -the benefits of full-fledged threads, though, requires that all memory -be shared (otherwise the OS has to tweak the page tables whenever it -switches contexts between your threads). - -> At their core, threads are a context switching efficiency tweak. - -This is the heart of the matter. Context switching is an operating -system problem, and *that* is where the optimization belongs. Threads -exist in large part because operating system vendors didn't bother to -do a good job of optimizing process context switching and -creation/destruction. - -Under Linux, from what I've read, process creation/destruction and -context switching happens almost as fast as thread context switching -on other operating systems (Windows in particular, if I'm not -mistaken). - -> Since DB's switch context a lot under many circumstances, threads -> should be a win under such circumstances. At the least, it should be -> helpful in situations where we have multiple CPUs to split query -> execution between. - -This is true, but I see little reason that we can't do the same thing -using fork()ed processes and shared memory instead. - -There is context switching within databases, to be sure, but I think -you'll be hard pressed to demonstrate that it is anything more than an -insignificant fraction of the total overhead incurred by the database. -I strongly suspect that much larger gains are to be had by optimizing -other areas of the database, such as the planner, the storage manager -(using mmap for file handling may prove useful here), the shared -memory system (mmap may be faster than System V style shared memory), -etc. - -The big overhead in the process model on most platforms is in creation -and destruction of processes. PostgreSQL has a relatively high -connection startup cost. But there are ways of dealing with this -problem other than threading, namely the use of a connection caching -middleware layer. Such layers exist for databases other than -PostgreSQL, so the high cost of fielding and setting up a database -connection is *not* unique to PostgreSQL ... which suggests that while -threading may help, it doesn't help *enough*. - -I'd rather see some development work go into a connection caching -process that understands the PostgreSQL wire protocol well enough to -look like a PostgreSQL backend to connecting processes, rather than -see a much larger amount of effort be spent on converting PostgreSQL -to a threaded architecture (and then discover that connection caching -is still needed anyway). - -> Certainly it's =possible= that threads have nothing to offer -> PostgreSQL, but IMHO it's not =probable=. Just another thing for me -> to add to my TODO heap for looking at... - -It's not that threads don't have anything to offer. It's that the -costs associated with them are high enough that it's not at all clear -that they're an overall win. - - --- -Kevin Brown kevin@sysexperts.com - - ----------------------------(end of broadcast)--------------------------- -TIP 6: Have you searched our list archives? - -http://archives.postgresql.org - -From pgsql-hackers-owner+M37876@postgresql.org Sat Apr 12 06:56:17 2003 -Return-path: -Received: from relay3.pgsql.com (relay3.pgsql.com [64.117.224.149]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h3CAuDS20700 - for ; Sat, 12 Apr 2003 06:56:15 -0400 (EDT) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by relay3.pgsql.com (Postfix) with ESMTP - id 35797EA81FF; Sat, 12 Apr 2003 10:55:59 +0000 (GMT) -X-Original-To: pgsql-hackers@postgresql.org -Received: from spampd.localdomain (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id 7393E4762EF - for ; Sat, 12 Apr 2003 06:54:48 -0400 (EDT) -Received: from filer (12-234-86-219.client.attbi.com [12.234.86.219]) - by postgresql.org (Postfix) with ESMTP id 423294762E1 - for ; Sat, 12 Apr 2003 06:54:44 -0400 (EDT) -Received: from localhost (localhost [127.0.0.1]) - (uid 1000) - by filer with local; Sat, 12 Apr 2003 03:54:52 -0700 -Date: Sat, 12 Apr 2003 03:54:52 -0700 -From: Kevin Brown -To: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Anyone working on better transaction locking? -Message-ID: <20030412105452.GV1833@filer> -Mail-Followup-To: Kevin Brown , - pgsql-hackers@postgresql.org -References: <20030409170926.GH2255@libertyrms.info> <20030411213259.GU1833@filer> <200304121221.12377.shridhar_daithankar@nospam.persistent.co.in> -MIME-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -Content-Transfer-Encoding: 7bit -Content-Disposition: inline -In-Reply-To: <200304121221.12377.shridhar_daithankar@nospam.persistent.co.in> -User-Agent: Mutt/1.4i -Organization: Frobozzco International -X-Spam-Status: No, hits=-39.4 required=5.0 - tests=BAYES_01,EMAIL_ATTRIBUTION,IN_REP_TO,QUOTED_EMAIL_TEXT, - QUOTE_TWICE_1,REFERENCES,REPLY_WITH_QUOTES,USER_AGENT_MUTT - autolearn=ham version=2.50 -X-Spam-Checker-Version: SpamAssassin 2.50 (1.173-2003-02-20-exp) -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -Shridhar Daithankar wrote: -> Apache does too many things to be a speed daemon and what it offers -> is pretty impressive from performance POV. -> -> But database is not webserver. It is not suppose to handle tons of -> concurrent requests. That is a fundamental difference. - -I'm not sure I necessarily agree with this. A database is just a -tool, a means of reliably storing information in such a way that it -can be retrieved quickly. Whether or not it "should" handle lots of -concurrent requests is a question that the person trying to use it -must answer. - -A better answer is that a database engine that can handle lots of -concurrent requests can also handle a smaller number, but not vice -versa. So it's clearly an advantage to have a database engine that -can handle lots of concurrent requests because such an engine can be -applied to a larger number of problems. That is, of course, assuming -that all other things are equal... - -There are situations in which a database would have to handle a lot of -concurrent requests. Handling ATM transactions over a large area is -one such situation. A database with current weather information might -be another, if it is actively queried by clients all over the country. -Acting as a mail store for a large organization is another. And, of -course, acting as a filesystem is definitely another. :-) - -> Well. Threading does not necessarily imply one thread per connection -> model. Threading can be used to make CPU work during I/O and taking -> advantage of SMP for things like sort etc. This is especially true -> for 2.4.x linux kernels where async I/O can not be used for threaded -> apps. as threads and signal do not mix together well. - -This is true, but whether you choose to limit the use of threads to a -few specific situations or use them throughout the database, the -dangers and difficulties faced by the developers when using threads -will be the same. - -> One connection per thread is not a good model for postgresql since -> it has already built a robust product around process paradigm. If I -> have to start a new database project today, a mix of process+thread -> is what I would choose bu postgresql is not in same stage of life. - -Certainly there are situations for which it would be advantageous to -have multiple concurrent actions happening on behalf of a single -connection, as you say. But that doesn't automatically mean that a -thread is the best overall solution. On systems such as Linux that -have fast process handling, processes are almost certainly the way to -go. On other systems such as Solaris or Windows, threads might be the -right answer (on Windows they might be the *only* answer). But my -argument here is simple: the responsibility of optimizing process -handling belongs to the maintainers of the OS. Application developers -shouldn't have to worry about this stuff. - -Of course, back here in the real world they *do* have to worry about -this stuff, and that's why it's important to quantify the problem. -It's not sufficient to say that "processes are slow and threads are -fast". Processes on the target platform may well be slow relative to -other systems (and relative to threads). But the question is: for the -problem being solved, how much overhead does process handling -represent relative to the total amount of overhead the solution itself -incurs? - -For instance, if we're talking about addressing the problem of -distributing sorts across multiple CPUs, the amount of overhead -involved in doing disk activity while sorting could easily swamp, in -the typical case, the overhead involved in creating parallel processes -to do the sorts themselves. And if that's the case, you may as well -gain the benefits of using full-fledged processes rather than deal -with the problems that come with the use of threads -- because the -gains to be found by using threads will be small in relative terms. - -> > > At their core, threads are a context switching efficiency tweak. -> > -> > This is the heart of the matter. Context switching is an operating -> > system problem, and *that* is where the optimization belongs. Threads -> > exist in large part because operating system vendors didn't bother to -> > do a good job of optimizing process context switching and -> > creation/destruction. -> -> But why would a database need a tons of context switches if it is -> not supposed to service loads to request simaltenously? If there are -> 50 concurrent connections, how much context switching overhead is -> involved regardless of amount of work done in a single connection? -> Remeber that database state is maintened in shared memory. It does -> not take a context switch to access it. - -If there are 50 concurrent connections with one process per -connection, then there are 50 database processes. The context switch -overhead is incurred whenever the current process blocks (or exhausts -its time slice) and the OS activates a different process. Since -database handling is generally rather I/O intensive as services go, -relatively few of those 50 processes are likely to be in a runnable -state, so I would expect the overall hit from context switching to be -rather low -- I'd expect the I/O subsystem to fall over well before -context switching became a real issue. - -Of course, all of that is independent of whether or not the database -can handle a lot of simultaneous requests. - -> > Under Linux, from what I've read, process creation/destruction and -> > context switching happens almost as fast as thread context switching -> > on other operating systems (Windows in particular, if I'm not -> > mistaken). -> -> I hear solaris also has very heavy processes. But postgresql has -> other issues with solaris as well. - -Yeah, I didn't want to mention Solaris because I haven't kept up with -it and thought that perhaps they had fixed this... - - --- -Kevin Brown kevin@sysexperts.com - - ----------------------------(end of broadcast)--------------------------- -TIP 2: you can get off all lists at once with the unregister command - (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) - -From pgsql-hackers-owner+M37883@postgresql.org Sat Apr 12 16:09:19 2003 -Return-path: -Received: from relay1.pgsql.com (relay1.pgsql.com [64.49.215.129]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id h3CK9HS03520 - for ; Sat, 12 Apr 2003 16:09:18 -0400 (EDT) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by relay1.pgsql.com (Postfix) with ESMTP - id 507626F768B; Sat, 12 Apr 2003 16:09:01 -0400 (EDT) -X-Original-To: pgsql-hackers@postgresql.org -Received: from spampd.localdomain (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id 06543475AE4 - for ; Sat, 12 Apr 2003 16:08:03 -0400 (EDT) -Received: from mail.gmx.net (mail.gmx.net [213.165.65.60]) - by postgresql.org (Postfix) with SMTP id C6DC347580B - for ; Sat, 12 Apr 2003 16:08:01 -0400 (EDT) -Received: (qmail 31386 invoked by uid 65534); 12 Apr 2003 20:08:13 -0000 -Received: from chello062178186201.1.15.tuwien.teleweb.at (EHLO beeblebrox) (62.178.186.201) - by mail.gmx.net (mp001-rz3) with SMTP; 12 Apr 2003 22:08:13 +0200 -Message-ID: <01cc01c3012f$526aaf80$3201a8c0@beeblebrox> -From: "Michael Paesold" -To: "Neil Conway" , "Kevin Brown" -cc: "PostgreSQL Hackers" -References: <20030409170926.GH2255@libertyrms.info> <20030411213259.GU1833@filer> <1050175777.392.13.camel@tokyo> -Subject: Re: [HACKERS] Anyone working on better transaction locking? -Date: Sat, 12 Apr 2003 22:08:40 +0200 -MIME-Version: 1.0 -Content-Type: text/plain; - charset="Windows-1252" -Content-Transfer-Encoding: 7bit -X-Priority: 3 -X-MSMail-Priority: Normal -X-Mailer: Microsoft Outlook Express 6.00.2800.1106 -X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 -X-Spam-Status: No, hits=-25.8 required=5.0 - tests=BAYES_20,EMAIL_ATTRIBUTION,QUOTED_EMAIL_TEXT,REFERENCES, - REPLY_WITH_QUOTES - autolearn=ham version=2.50 -X-Spam-Checker-Version: SpamAssassin 2.50 (1.173-2003-02-20-exp) -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -Neil Conway wrote: - -> Furthermore, IIRC PostgreSQL's relatively slow connection creation time -> has as much to do with other per-backend initialization work as it does -> with the time to actually fork() a new backend. If there is interest in -> optimizing backend startup time, my guess would be that there is plenty -> of room for improvement without requiring the replacement of processes -> with threads. - -I see there is a whole TODO Chapter devoted to the topic. There is the idea -of pre-forked and persistent backends. That would be very useful in an -environment where it's quite hard to use connection pooling. We are -currently working on a mail system for a free webmail. The mda (mail -delivery agent) written in C connects to the pg database to do some queries -everytime a new mail comes in. I didn't find a solution for connection -pooling yet. - -About the TODO items, apache has a nice description of their accept() -serialization: -http://httpd.apache.org/docs-2.0/misc/perf-tuning.html - -Perhaps this could be useful if someone decided to start implementing those -features. - -Regards, -Michael Paesold - - ----------------------------(end of broadcast)--------------------------- -TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org - diff --git a/doc/TODO.detail/timezone b/doc/TODO.detail/timezone deleted file mode 100644 index f4fabc16b5..0000000000 --- a/doc/TODO.detail/timezone +++ /dev/null @@ -1,2829 +0,0 @@ -From pgsql-hackers-owner+M60207=pgman=candle.pha.pa.us@postgresql.org Thu Oct 21 07:25:41 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9LBPdf26430 - for ; Thu, 21 Oct 2004 07:25:40 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 0548B32A593 - for ; Thu, 21 Oct 2004 12:25:37 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 74257-06 for ; - Thu, 21 Oct 2004 11:25:33 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id B0DB532A544 - for ; Thu, 21 Oct 2004 12:25:36 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 1124A329FAB - for ; Thu, 21 Oct 2004 12:23:00 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 70900-09 - for ; - Thu, 21 Oct 2004 11:22:43 +0000 (GMT) -Received: from zigo.dhs.org (as2-4-3.an.g.bonet.se [194.236.34.191]) - by svr1.postgresql.org (Postfix) with ESMTP id 301C532A301 - for ; Thu, 21 Oct 2004 12:22:43 +0100 (BST) -Received: from zigo.zigo.dhs.org (zigo.zigo.dhs.org [192.168.0.1]) - by zigo.dhs.org (Postfix) with ESMTP id A022C8467 - for ; Thu, 21 Oct 2004 13:22:40 +0200 (CEST) -Date: Thu, 21 Oct 2004 13:22:40 +0200 (CEST) -From: Dennis Bjorklund -To: pgsql-hackers@postgresql.org -Subject: [HACKERS] timestamp with time zone a la sql99 -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=ISO-8859-1 -Content-Transfer-Encoding: 8BIT -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -I've made a partial implementation of a datatype "timestamp with time -zone" as described in the sql standard. The current type "timestamptz" -does not store the time zone as a standard one should do. So I've made a -new type I've called timestampstdtz that does store the time zone as the -standard demands. - -Let me show a bit of what currently works in my implementation: - - dennis=# CREATE TABLE foo ( - a timestampstdtz, - - primary key (a) - ); - dennis=# INSERT INTO foo VALUES ('1993-02-04 13:00 UTC'); - dennis=# INSERT INTO foo VALUES ('1999-06-01 14:00 CET'); - dennis=# INSERT INTO foo VALUES ('2003-08-21 15:00 PST'); - - dennis=# SELECT a FROM foo; - a - ------------------------ - 1993-02-04 13:00:00+00 - 1999-06-01 14:00:00+01 - 2003-08-21 15:00:00-08 - - dennis=# SELECT a AT TIME ZONE 'CET' FROM foo; - timezone - ------------------------ - 1993-02-04 14:00:00+01 - 1999-06-01 14:00:00+01 - 2003-08-22 00:00:00+01 - -My plan is to make a GUC variable so that one can tell PG that constructs -like "timestamp with time zone" will map to timestampstdtz instead of -timestamptz (some old databases might need the old so unless we want to -break old code this is the easiest solution I can find). - -I've made an implicit cast from timestampstdtz to timestamptz that just -forgets about the time zone. In the other direction I've made an -assignment cast that make a timestamp with time zone 0 (that's what a -timestamptz is anyway). Would it be possible to make it implicit in both -directions? I currently don't think that you want that, but is it -possible? - -With the implicit cast in place I assume it would be safe to change -functions like now() to return a timestampstdtz? I've not tried yet but I -will. As far as I can tell the cast would make old code that use now() to -still work as before. - -Any comments before I invest more time into this subject? - --- -/Dennis Björklund - - ----------------------------(end of broadcast)--------------------------- -TIP 2: you can get off all lists at once with the unregister command - (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) - -From pgsql-hackers-owner+M60208=pgman=candle.pha.pa.us@postgresql.org Thu Oct 21 10:34:36 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9LEYYf08049 - for ; Thu, 21 Oct 2004 10:34:35 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 34B0F32A2AB - for ; Thu, 21 Oct 2004 15:34:30 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 38287-03 for ; - Thu, 21 Oct 2004 14:34:26 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 6A38132A1B6 - for ; Thu, 21 Oct 2004 15:34:29 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 535FE32A9EE - for ; Thu, 21 Oct 2004 15:29:17 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 34535-09 - for ; - Thu, 21 Oct 2004 14:29:10 +0000 (GMT) -Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) - by svr1.postgresql.org (Postfix) with ESMTP id 41E4F32A9D6 - for ; Thu, 21 Oct 2004 15:29:12 +0100 (BST) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id i9LET86O015233; - Thu, 21 Oct 2004 10:29:10 -0400 (EDT) -To: Dennis Bjorklund -cc: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: -References: -Comments: In-reply-to Dennis Bjorklund - message dated "Thu, 21 Oct 2004 13:22:40 +0200" -Date: Thu, 21 Oct 2004 10:29:07 -0400 -Message-ID: <15232.1098368947@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -Dennis Bjorklund writes: -> I've made a partial implementation of a datatype "timestamp with time -> zone" as described in the sql standard. The current type "timestamptz" -> does not store the time zone as a standard one should do. - -I'm aware that there are aspects of the spec behavior that appear to -require that, but is it really an improvement over the implementation -we have? This is an area in which the standard is pretty brain-dead ---- the entire concept of a "time with time zone" datatype is rather -suspect, for instance. - -In particular, I wonder how you will handle daylight-savings issues. -The spec definition seems to preclude doing anything intelligent with -DST, as they equate a timezone with a fixed offset from UTC. That's -not how it works in (large parts of) the real world. - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 6: Have you searched our list archives? - - http://archives.postgresql.org - -From pgsql-hackers-owner+M60210=pgman=candle.pha.pa.us@postgresql.org Thu Oct 21 11:08:02 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9LF7xf13992 - for ; Thu, 21 Oct 2004 11:08:00 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id AEBEE32AB8E - for ; Thu, 21 Oct 2004 16:07:55 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 51273-03 for ; - Thu, 21 Oct 2004 15:07:55 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 506C832AB6A - for ; Thu, 21 Oct 2004 16:07:55 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 59130329F96 - for ; Thu, 21 Oct 2004 16:02:14 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 48602-06 - for ; - Thu, 21 Oct 2004 15:01:59 +0000 (GMT) -Received: from zigo.dhs.org (as2-4-3.an.g.bonet.se [194.236.34.191]) - by svr1.postgresql.org (Postfix) with ESMTP id 8ED0932A095 - for ; Thu, 21 Oct 2004 16:01:54 +0100 (BST) -Received: from zigo.zigo.dhs.org (zigo.zigo.dhs.org [192.168.0.1]) - by zigo.dhs.org (Postfix) with ESMTP - id D08B88467; Thu, 21 Oct 2004 17:01:52 +0200 (CEST) -Date: Thu, 21 Oct 2004 17:01:52 +0200 (CEST) -From: Dennis Bjorklund -To: Tom Lane -cc: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: <15232.1098368947@sss.pgh.pa.us> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=ISO-8859-1 -Content-Transfer-Encoding: 8BIT -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -On Thu, 21 Oct 2004, Tom Lane wrote: - -> > I've made a partial implementation of a datatype "timestamp with time -> > zone" as described in the sql standard. The current type "timestamptz" -> > does not store the time zone as a standard one should do. -> -> I'm aware that there are aspects of the spec behavior that appear to -> require that, but is it really an improvement over the implementation -> we have? - -Improvement and improvement. The actual time value is of course the same -(the utc part of a timestamp) and the only thing extra you get is that the -time zone is stored. The extra information you do have now, when stored in -this way, is that you store both a utc time and a local time. Will any -application ever need that? Who knows? I think it makes sense and is an -easier model to think about then what pg uses today. So I would use it -even if it means using 2 bytes more storage then what timestamptz do - -Just that it is standard also makes it useful. The more things of the -standard we support the easier it is to move between databases. This is -important to me. - -I also want to make a general statement that I think that whenever we use -standard syntax we should give it a standard semantics. I don't mind -extensions at all, but as much as we can we should make sure that they -don't clash with standard syntax and semantics. - -> This is an area in which the standard is pretty brain-dead -> --- the entire concept of a "time with time zone" datatype is rather -> suspect, for instance. - -I havn't look that much at "time with time zone" yet, just timestamps. - -I can't see why time with time zone should not also be supported. I can't -really imagine it being used without a date, but if someone wants to store -timestamps as a date+time with time zone, then why not. It would be extra -work tu is it instead of a timestamp (especially for cases where the time -wraps over to the prev/next day), but hey. - -> In particular, I wonder how you will handle daylight-savings issues. -> The spec definition seems to preclude doing anything intelligent with -> DST, as they equate a timezone with a fixed offset from UTC. That's -> not how it works in (large parts of) the real world. - -The tz in the standard is a offset from utc, yes. So when you store a -value you tell it what offset you use. If you are using daylight-savings -time it might be +02 and if not dst it might be +01. What else would you -want to do with it? It's not like you can do anything else with it in pg -as of today, can you? - -The stored tz does not say what region of the globe you are in, it says -the distance away from utc in minutes that you are. I could imagine -another datatype that stores the time zone as name, but that's not what -timestamp with time zone does. - --- -/Dennis Björklund - - ----------------------------(end of broadcast)--------------------------- -TIP 8: explain analyze is your friend - -From pgsql-hackers-owner+M60232=pgman=candle.pha.pa.us@postgresql.org Fri Oct 22 08:43:06 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9MCh4f16646 - for ; Fri, 22 Oct 2004 08:43:04 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id D389532A4BA - for ; Fri, 22 Oct 2004 13:42:58 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 01914-09 for ; - Fri, 22 Oct 2004 12:42:51 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 82BD832A4B3 - for ; Fri, 22 Oct 2004 13:42:58 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 05C2132A3F5 - for ; Fri, 22 Oct 2004 13:39:37 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 03281-02 - for ; - Fri, 22 Oct 2004 12:39:20 +0000 (GMT) -Received: from lakermmtao07.cox.net (lakermmtao07.cox.net [68.230.240.32]) - by svr1.postgresql.org (Postfix) with ESMTP id 1971732A32B - for ; Fri, 22 Oct 2004 13:39:26 +0100 (BST) -Received: from [192.168.0.9] (really [24.250.237.182]) - by lakermmtao07.cox.net - (InterMail vM.6.01.03.04 201-2131-111-106-20040729) with ESMTP - id <20041022123912.IJSP14063.lakermmtao07.cox.net@[192.168.0.9]>; - Fri, 22 Oct 2004 08:39:12 -0400 -From: Robert Treat -To: Dennis Bjorklund -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -Date: Fri, 22 Oct 2004 08:37:33 -0400 -User-Agent: KMail/1.6.2 -cc: Tom Lane , pgsql-hackers@postgresql.org -References: -In-Reply-To: -MIME-Version: 1.0 -Content-Disposition: inline -Content-Type: text/plain; - charset="iso-8859-1" -Content-Transfer-Encoding: 7bit -Message-ID: <200410220837.33886.xzilla@users.sourceforge.net> -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -On Thursday 21 October 2004 11:01, Dennis Bjorklund wrote: -> On Thu, 21 Oct 2004, Tom Lane wrote: -> > I'm aware that there are aspects of the spec behavior that appear to -> > require that, but is it really an improvement over the implementation -> > we have? -> -> Improvement and improvement. The actual time value is of course the same -> (the utc part of a timestamp) and the only thing extra you get is that the -> time zone is stored. The extra information you do have now, when stored in -> this way, is that you store both a utc time and a local time. Will any -> application ever need that? Who knows? I think it makes sense and is an -> easier model to think about then what pg uses today. So I would use it -> even if it means using 2 bytes more storage then what timestamptz do -> - -In a fit of early morning, pre-coffee thoughts, I'm thinking this might be -just what I've been looking for. In one of my apps we take calls from around -the country for customers and store the time that call came in. Unfortunately -we need to know things like how many calls did we take in an hour across -customers, but also how many calls did we take at 6AM local time to the -customer. The way PostgreSQL works now, you have to store some extra bits -of info in another column and then reassemble it to be able to determine -those two queries, but it sounds like your timestampstdtz would allow that -information to be stored together, as it should be. - --- -Robert Treat -Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL - ----------------------------(end of broadcast)--------------------------- -TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org - -From pgsql-hackers-owner+M60235=pgman=candle.pha.pa.us@postgresql.org Fri Oct 22 10:18:44 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9MEIhf03000 - for ; Fri, 22 Oct 2004 10:18:44 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 63ACB329FE3 - for ; Fri, 22 Oct 2004 15:18:38 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 35128-09 for ; - Fri, 22 Oct 2004 14:18:30 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 3439A329E73 - for ; Fri, 22 Oct 2004 15:18:38 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id C332832A5AA - for ; Fri, 22 Oct 2004 15:13:18 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 32986-06 - for ; - Fri, 22 Oct 2004 14:13:09 +0000 (GMT) -Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) - by svr1.postgresql.org (Postfix) with ESMTP id E7E6F32A576 - for ; Fri, 22 Oct 2004 15:13:16 +0100 (BST) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id i9MEDIA4006541; - Fri, 22 Oct 2004 10:13:18 -0400 (EDT) -To: Robert Treat -cc: Dennis Bjorklund , pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: <200410220837.33886.xzilla@users.sourceforge.net> -References: <200410220837.33886.xzilla@users.sourceforge.net> -Comments: In-reply-to Robert Treat - message dated "Fri, 22 Oct 2004 08:37:33 -0400" -Date: Fri, 22 Oct 2004 10:13:18 -0400 -Message-ID: <6540.1098454398@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -Robert Treat writes: -> In a fit of early morning, pre-coffee thoughts, I'm thinking this might be -> just what I've been looking for. In one of my apps we take calls from around -> the country for customers and store the time that call came in. Unfortunately -> we need to know things like how many calls did we take in an hour across -> customers, but also how many calls did we take at 6AM local time to the -> customer. The way PostgreSQL works now, you have to store some extra bits -> of info in another column and then reassemble it to be able to determine -> those two queries, but it sounds like your timestampstdtz would allow that -> information to be stored together, as it should be. - -As far as I can tell, Dennis is planning slavish adherence to the spec, -which will mean that the datatype is unable to cope effectively with -daylight-savings issues. So I'm unconvinced that it will be very -helpful to you for remembering local time in addition to true -(universal) time. - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate - subscribe-nomail command to majordomo@postgresql.org so that your - message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M60237=pgman=candle.pha.pa.us@postgresql.org Fri Oct 22 10:33:04 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9MEX3f05134 - for ; Fri, 22 Oct 2004 10:33:03 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id AB3B432A907 - for ; Fri, 22 Oct 2004 15:32:57 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 42366-04 for ; - Fri, 22 Oct 2004 14:32:49 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 28B5E32A6BE - for ; Fri, 22 Oct 2004 15:32:56 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 5366A32A923 - for ; Fri, 22 Oct 2004 15:28:12 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 41328-02 - for ; - Fri, 22 Oct 2004 14:28:03 +0000 (GMT) -Received: from zigo.dhs.org (as2-4-3.an.g.bonet.se [194.236.34.191]) - by svr1.postgresql.org (Postfix) with ESMTP id 0277A32A916 - for ; Fri, 22 Oct 2004 15:28:10 +0100 (BST) -Received: from zigo.zigo.dhs.org (zigo.zigo.dhs.org [192.168.0.1]) - by zigo.dhs.org (Postfix) with ESMTP - id 336C88467; Fri, 22 Oct 2004 16:28:12 +0200 (CEST) -Date: Fri, 22 Oct 2004 16:28:12 +0200 (CEST) -From: Dennis Bjorklund -To: Tom Lane -cc: Robert Treat , - -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: <6540.1098454398@sss.pgh.pa.us> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=ISO-8859-1 -Content-Transfer-Encoding: 8BIT -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -On Fri, 22 Oct 2004, Tom Lane wrote: - -> As far as I can tell, Dennis is planning slavish adherence to the spec, -> which will mean that the datatype is unable to cope effectively with -> daylight-savings issues. So I'm unconvinced that it will be very -> helpful to you for remembering local time in addition to true -> (universal) time. - -And exactly what issues is it that you see? The only thing I can think of -is if you have a timestamp and then add an interval to it so we jump past -the daylight saving time change date. Then the new timestamp will keep the -old timezone data of say +01 even though we now have jumped into the -daylight saving period of +02. - -If you are just storing actual timestamps then the standard definition -works just fine. If I store '2004-10-22 16:20:04 +02' then that's exactly -what I get back. No problem what so ever. There is no DST problem with -that. - -It's possible that I will introduce some daylight saving bit or something -like that, I'm not sure yet and I will not commit to anything until I've -thought it over. I don't think there are that much of a problem as you -claim however. Could you give a concret example where it will be a -problem? - -My current thinking is that storing the time zone value as HH:MM is -just fine and you avoid all the problems with political changes of when -the DST is in effect or not. - --- -/Dennis Björklund - - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate - subscribe-nomail command to majordomo@postgresql.org so that your - message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M60240=pgman=candle.pha.pa.us@postgresql.org Fri Oct 22 10:58:26 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9MEwOf10545 - for ; Fri, 22 Oct 2004 10:58:24 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 8F3BC32A0AC - for ; Fri, 22 Oct 2004 15:58:18 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 53613-02 for ; - Fri, 22 Oct 2004 14:58:10 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 32CD532A0A2 - for ; Fri, 22 Oct 2004 15:58:18 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 65D9932A0AC - for ; Fri, 22 Oct 2004 15:54:30 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 49578-08 - for ; - Fri, 22 Oct 2004 14:54:12 +0000 (GMT) -Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) - by svr1.postgresql.org (Postfix) with ESMTP id A7D2A329FB7 - for ; Fri, 22 Oct 2004 15:54:17 +0100 (BST) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id i9MEsJsB006995; - Fri, 22 Oct 2004 10:54:19 -0400 (EDT) -To: Dennis Bjorklund -cc: Robert Treat , pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: -References: -Comments: In-reply-to Dennis Bjorklund - message dated "Fri, 22 Oct 2004 16:28:12 +0200" -Date: Fri, 22 Oct 2004 10:54:19 -0400 -Message-ID: <6994.1098456859@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -Dennis Bjorklund writes: -> And exactly what issues is it that you see? The only thing I can think of -> is if you have a timestamp and then add an interval to it so we jump past -> the daylight saving time change date. Then the new timestamp will keep the -> old timezone data of say +01 even though we now have jumped into the -> daylight saving period of +02. - -Isn't that sufficient? You can't design a datatype by thinking only of -the data values it stores; you have to think about the operations you -intend to provide as well. A non-DST-capable timestamp datatype is -inherently a few bricks shy of a load. (BTW we really need to fix -the interval type as well...) - -At bottom, what I want to be able to do is say - '2004-10-22 10:50:16.916003 America/New_York' -and have the datatype preserve *all* of the information in that. You -are complaining because the existing type only remembers the equivalent -universal time and not the timezone spec. Why should I be satisfied if -it stores only the GMT offset and not the knowledge of which timezone -this really is? - -> My current thinking is that storing the time zone value as HH:MM is -> just fine and you avoid all the problems with political changes of when -> the DST is in effect or not. - -This is fundamentally misguided. Time zones *are* political whether you -like it or not, and people *do* expect DST-awareness whether you like it -or not. If you still use any computer systems that need to be reset -twice a year because their designers thought DST was not their problem, -don't you roundly curse them every time you have to do it? - -If you were planning to store a real (potentially DST-aware) timezone -spec in the data values, I'd be happy. But storing a fixed GMT offset -is going to be a step backwards compared to existing functionality. The -fact that it's sufficient to satisfy the DST-ignorant SQL spec does not -make it a reasonable design for the real world. - -One way to do this would be to create a system catalog with entries for -all known timezones, and then represent timestamptz values as universal -time plus an OID from that catalog. There are other ways that small -integer codes could be mapped to timezones of course. - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 2: you can get off all lists at once with the unregister command - (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) - -From pgsql-hackers-owner+M60239=pgman=candle.pha.pa.us@postgresql.org Fri Oct 22 10:49:13 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9MEnCf08566 - for ; Fri, 22 Oct 2004 10:49:12 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 5B4D432A0A2 - for ; Fri, 22 Oct 2004 15:49:07 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 50773-01 for ; - Fri, 22 Oct 2004 14:48:59 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 230EA329F96 - for ; Fri, 22 Oct 2004 15:49:07 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 7CCE332A20E - for ; Fri, 22 Oct 2004 15:46:10 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 45902-10 - for ; - Fri, 22 Oct 2004 14:45:52 +0000 (GMT) -Received: from wolff.to (wolff.to [66.93.249.74]) - by svr1.postgresql.org (Postfix) with SMTP id 30F0F32A1FA - for ; Fri, 22 Oct 2004 15:45:58 +0100 (BST) -Received: (qmail 17526 invoked by uid 500); 22 Oct 2004 14:56:44 -0000 -Date: Fri, 22 Oct 2004 09:56:44 -0500 -From: Bruno Wolff III -To: Dennis Bjorklund -cc: Tom Lane , Robert Treat , - pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -Message-ID: <20041022145644.GA17238@wolff.to> -Mail-Followup-To: Dennis Bjorklund , - Tom Lane , - Robert Treat , - pgsql-hackers@postgresql.org -References: <6540.1098454398@sss.pgh.pa.us> -MIME-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -In-Reply-To: -User-Agent: Mutt/1.5.6i -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -On Fri, Oct 22, 2004 at 16:28:12 +0200, - Dennis Bjorklund wrote: -> On Fri, 22 Oct 2004, Tom Lane wrote: -> -> > As far as I can tell, Dennis is planning slavish adherence to the spec, -> > which will mean that the datatype is unable to cope effectively with -> > daylight-savings issues. So I'm unconvinced that it will be very -> > helpful to you for remembering local time in addition to true -> > (universal) time. -> -> And exactly what issues is it that you see? The only thing I can think of -> is if you have a timestamp and then add an interval to it so we jump past -> the daylight saving time change date. Then the new timestamp will keep the -> old timezone data of say +01 even though we now have jumped into the -> daylight saving period of +02. - -I think for just storing values you are fine. When it comes to adding or -subtracting intervals you might get some unexpected results. - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate - subscribe-nomail command to majordomo@postgresql.org so that your - message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M60253=pgman=candle.pha.pa.us@postgresql.org Fri Oct 22 14:44:05 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9MIi3f14607 - for ; Fri, 22 Oct 2004 14:44:03 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id B336CEAEDAF - for ; Fri, 22 Oct 2004 19:43:39 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 21148-08 for ; - Fri, 22 Oct 2004 18:43:46 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 4FC1AEAEDAB - for ; Fri, 22 Oct 2004 19:43:39 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 178B1EAEF44 - for ; Fri, 22 Oct 2004 19:38:04 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 20287-01 - for ; - Fri, 22 Oct 2004 18:38:00 +0000 (GMT) -Received: from www.postgresql.com (www.postgresql.com [200.46.204.209]) - by svr1.postgresql.org (Postfix) with ESMTP id D1F79EAEE4F - for ; Fri, 22 Oct 2004 19:37:45 +0100 (BST) -Received: from zigo.dhs.org (as2-4-3.an.g.bonet.se [194.236.34.191]) - by www.postgresql.com (Postfix) with ESMTP id 358575A1CED - for ; Fri, 22 Oct 2004 16:34:33 +0100 (BST) -Received: from zigo.zigo.dhs.org (zigo.zigo.dhs.org [192.168.0.1]) - by zigo.dhs.org (Postfix) with ESMTP - id 89B7E8467; Fri, 22 Oct 2004 17:34:19 +0200 (CEST) -Date: Fri, 22 Oct 2004 17:34:19 +0200 (CEST) -From: Dennis Bjorklund -To: Tom Lane -cc: Robert Treat , - -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: <6994.1098456859@sss.pgh.pa.us> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=ISO-8859-1 -Content-Transfer-Encoding: 8BIT -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -On Fri, 22 Oct 2004, Tom Lane wrote: - -> At bottom, what I want to be able to do is say -> '2004-10-22 10:50:16.916003 America/New_York' - -Yes, that's what we said in the last mail and I think there is a value in -having something like this. - -> universal time and not the timezone spec. Why should I be satisfied if -> it stores only the GMT offset and not the knowledge of which timezone -> this really is? - -You don't need to be satisfied with it. I think a type like the above -would be fine to have. It should however not be called "TIMESTAMP WITH -TIME ZONE" because there is already a definition of that type. We can not -hijack standard types. I would not mind a type like TIMESTAMP WITH TIME -ZONE NAME (or some other name). I could even imagine that I could -implement something like that one day. - -> > My current thinking is that storing the time zone value as HH:MM is -> > just fine and you avoid all the problems with political changes of when -> > the DST is in effect or not. -> -> This is fundamentally misguided. Time zones *are* political whether you -> like it or not, and people *do* expect DST-awareness whether you like it -> or not. - -And I never said that time zones are not political, just that HH:MM is a -usable approximation that works fairly well. - -> But storing a fixed GMT offset is going to be a step backwards compared -> to existing functionality. - -It's not a step backwards since you can do everything you can do with the -current type plus a little bit more. It's however not a step to the -datatype discussed above. - -> One way to do this would be to create a system catalog with entries for -> all known timezones, and then represent timestamptz values as universal -> time plus an OID from that catalog. There are other ways that small -> integer codes could be mapped to timezones of course. - -This is just fine. You try to make it sound like I am against such a -datatype, I am not. It's however not the datatype that we can expect -applications and other databases to use. So why should we settle for only -that type. Just because you can make a perfect datatype it doesn't mean -that the standard datatype should just be ignored. - -What would you store when the user supplies a timestamp like '2004-10-22 -17:21:00 +0200'. Should you reject that because you don't know the -time zone name? So your datatype will not work for applications that try -to be compatable with many databases by using the standard? - -Maybe one could make a datatype called TIMESTAMP WITH TIME ZONE that can -accept both HH:MM and TimeZoneName. Whenever you store values with HH:MM -time zones you will get the same problem when you add an interval as the -standard type has. - --- -/Dennis Björklund - - ----------------------------(end of broadcast)--------------------------- -TIP 8: explain analyze is your friend - -From pgsql-hackers-owner+M60266=pgman=candle.pha.pa.us@postgresql.org Fri Oct 22 17:04:46 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9ML4jf06676 - for ; Fri, 22 Oct 2004 17:04:45 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id C13F9EAE46B - for ; Fri, 22 Oct 2004 22:04:20 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 66740-03 for ; - Fri, 22 Oct 2004 21:04:28 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 75764EAE1A2 - for ; Fri, 22 Oct 2004 22:04:20 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 2E4AFEAE486 - for ; Fri, 22 Oct 2004 22:01:22 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 63290-09 - for ; - Fri, 22 Oct 2004 21:01:22 +0000 (GMT) -Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) - by svr1.postgresql.org (Postfix) with ESMTP id 88B3FEAE492 - for ; Fri, 22 Oct 2004 22:01:13 +0100 (BST) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id i9ML1WrB018877; - Fri, 22 Oct 2004 17:01:34 -0400 (EDT) -To: Dennis Bjorklund -cc: Robert Treat , pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: -References: -Comments: In-reply-to Dennis Bjorklund - message dated "Fri, 22 Oct 2004 17:34:19 +0200" -Date: Fri, 22 Oct 2004 17:01:32 -0400 -Message-ID: <18876.1098478892@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -Dennis Bjorklund writes: -> You don't need to be satisfied with it. I think a type like the above -> would be fine to have. It should however not be called "TIMESTAMP WITH -> TIME ZONE" because there is already a definition of that type. We can not -> hijack standard types. - -Sure we can, as long as they are upward compatible with the standard -behavior. The spec says you can put a numeric-GMT-offset zone in and -get a numeric-GMT-offset zone out. We can do that and also support -named, possibly DST-aware zones. This seems a whole lot better to me -than having two different types (the idea of a GUC variable to choose -which one is selected by a given type name is just horrid). - ->> But storing a fixed GMT offset is going to be a step backwards compared ->> to existing functionality. - -> It's not a step backwards since you can do everything you can do with the -> current type plus a little bit more. - -... except get useful answers from interval addition ... - -> What would you store when the user supplies a timestamp like '2004-10-22 -> 17:21:00 +0200'. Should you reject that because you don't know the -> time zone name? - -You are attacking a straw man. - -We have put a great deal of work into 8.0 to add the ability to support -real-world zones fully. We did not import src/timezone because we -needed it to implement the SQL spec; we did so because we needed it to -implement what real users want. We are not fully there yet (can't do AT -TIME ZONE conversions with all zones yet, for instance) but I am hoping -to be there by 8.1. It would be folly to invent a timestamp with time -zone type that is going in the other direction while we are trying to -bring the rest of the system up to full speed by allowing all timezone -kinds everywhere. - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate - subscribe-nomail command to majordomo@postgresql.org so that your - message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M60267=pgman=candle.pha.pa.us@postgresql.org Fri Oct 22 17:22:15 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9MLMEf09207 - for ; Fri, 22 Oct 2004 17:22:14 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 0B2D7EAE4AC - for ; Fri, 22 Oct 2004 22:21:49 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 70387-04 for ; - Fri, 22 Oct 2004 21:21:56 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 0930BEAE4AA - for ; Fri, 22 Oct 2004 22:21:48 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 292E8EAC955 - for ; Fri, 22 Oct 2004 22:18:05 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 68984-06 - for ; - Fri, 22 Oct 2004 21:18:04 +0000 (GMT) -Received: from zigo.dhs.org (as2-4-3.an.g.bonet.se [194.236.34.191]) - by svr1.postgresql.org (Postfix) with ESMTP id A3C4DEAE489 - for ; Fri, 22 Oct 2004 22:17:51 +0100 (BST) -Received: from zigo.zigo.dhs.org (zigo.zigo.dhs.org [192.168.0.1]) - by zigo.dhs.org (Postfix) with ESMTP - id 065508469; Fri, 22 Oct 2004 23:18:08 +0200 (CEST) -Date: Fri, 22 Oct 2004 23:18:07 +0200 (CEST) -From: Dennis Bjorklund -To: Tom Lane -cc: Robert Treat , - -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: <18876.1098478892@sss.pgh.pa.us> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=ISO-8859-1 -Content-Transfer-Encoding: 8BIT -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -On Fri, 22 Oct 2004, Tom Lane wrote: - -> than having two different types (the idea of a GUC variable to choose -> which one is selected by a given type name is just horrid). - -That is needed no matter what change you do if you want old programs that -use the current timestamp with time zone to work. Today you don't get back -the same time zone as you insert, programs might depend on that. - -> We are not fully there yet (can't do AT TIME ZONE conversions with all -> zones yet, for instance) - -Why is that? When one start with a utc value, performing a AT TIME ZONE -operation doesn't look so complicated. - --- -/Dennis Björklund - - ----------------------------(end of broadcast)--------------------------- -TIP 5: Have you checked our extensive FAQ? - - http://www.postgresql.org/docs/faqs/FAQ.html - -From pgsql-hackers-owner+M60268=pgman=candle.pha.pa.us@postgresql.org Fri Oct 22 17:41:36 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9MLfYf12294 - for ; Fri, 22 Oct 2004 17:41:35 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id EFAEAEAE4AA - for ; Fri, 22 Oct 2004 22:41:08 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 75185-08 for ; - Fri, 22 Oct 2004 21:41:16 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 311A7EAE4AF - for ; Fri, 22 Oct 2004 22:41:08 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id AC0CFEAE486 - for ; Fri, 22 Oct 2004 22:38:14 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 76579-03 - for ; - Fri, 22 Oct 2004 21:38:13 +0000 (GMT) -Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) - by svr1.postgresql.org (Postfix) with ESMTP id BD0AAEAD7E2 - for ; Fri, 22 Oct 2004 22:38:03 +0100 (BST) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id i9MLcOnT019118; - Fri, 22 Oct 2004 17:38:25 -0400 (EDT) -To: Dennis Bjorklund -cc: Robert Treat , pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: -References: -Comments: In-reply-to Dennis Bjorklund - message dated "Fri, 22 Oct 2004 23:18:07 +0200" -Date: Fri, 22 Oct 2004 17:38:24 -0400 -Message-ID: <19117.1098481104@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -Dennis Bjorklund writes: -> On Fri, 22 Oct 2004, Tom Lane wrote: ->> than having two different types (the idea of a GUC variable to choose ->> which one is selected by a given type name is just horrid). - -> That is needed no matter what change you do if you want old programs that -> use the current timestamp with time zone to work. Today you don't get back -> the same time zone as you insert, programs might depend on that. - -[ shrug... ] We've made much larger changes than that in the name of -standards compliance. In practice I think the majority of apps are -working in contexts where they will get back the same zone as they -inserted, if they inserted a zone explicitly at all, so the risk of -breakage is not that high. Having a GUC variable that changes the -semantics underneath you is *much* riskier, to judge by past experience. - ->> We are not fully there yet (can't do AT TIME ZONE conversions with all ->> zones yet, for instance) - -> Why is that? - -Because it's not done yet. There's a set of GMT-offset-only zone names -wired into the datetime code (look in the "datetime token table") and -those are what AT TIME ZONE knows how to deal with. We need to unify -that old stuff with the src/timezone code, but we ran out of time to do -it in 8.0. - -The way I see it, we have three sorts of zones to deal with: fixed -numeric offsets from UTC, names that represent fixed offsets (eg, "EST" -is the same as UTC-5), and names that represent DST-variable offsets -(eg, "EST5EDT"). For what are now entirely historical reasons, various -parts of the system cope with different subsets of these three types. -I want to get to a state where you can use any of them in any context -and it Just Works. (While we are at it, we need to make the set of -recognized zone names user-configurable; the australian_timezones kluge -satisfies our contributors Down Under, but there are a lot of unhappy -people still, because for instance IST means different things in Israel -and India.) - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 2: you can get off all lists at once with the unregister command - (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) - -From pgsql-hackers-owner+M60269=pgman=candle.pha.pa.us@postgresql.org Fri Oct 22 17:51:14 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9MLpCf14192 - for ; Fri, 22 Oct 2004 17:51:13 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 43D6BEAE489 - for ; Fri, 22 Oct 2004 22:50:47 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 78632-05 for ; - Fri, 22 Oct 2004 21:50:55 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id EFC45EAE486 - for ; Fri, 22 Oct 2004 22:50:46 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 42518EAC955 - for ; Fri, 22 Oct 2004 22:48:36 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 77399-07 - for ; - Fri, 22 Oct 2004 21:48:35 +0000 (GMT) -Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) - by svr1.postgresql.org (Postfix) with ESMTP id 8BB75EAE46B - for ; Fri, 22 Oct 2004 22:48:27 +0100 (BST) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id i9MLmm1l019192; - Fri, 22 Oct 2004 17:48:48 -0400 (EDT) -To: Dennis Bjorklund -cc: Robert Treat , pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: <19117.1098481104@sss.pgh.pa.us> -References: <19117.1098481104@sss.pgh.pa.us> -Comments: In-reply-to Tom Lane - message dated "Fri, 22 Oct 2004 17:38:24 -0400" -Date: Fri, 22 Oct 2004 17:48:48 -0400 -Message-ID: <19191.1098481728@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - ->> That is needed no matter what change you do if you want old programs that ->> use the current timestamp with time zone to work. Today you don't get back ->> the same time zone as you insert, programs might depend on that. - -> [ shrug... ] We've made much larger changes than that in the name of -> standards compliance. - -BTW, even if you do want output like that, that doesn't make two -datatypes a good idea. It'd be better to add a couple of DateStyle-like -formatting options: - * rotate all timestamps into current TimeZone for display, or not; - * display the timezone numerically, or as originally given. - -A DateStyle kind of GUC variable is a lot less dangerous than what you -were proposing, because getting it wrong doesn't mean you have the wrong -data stored in the database ... - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M60274=pgman=candle.pha.pa.us@postgresql.org Sat Oct 23 02:11:58 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9N6Buf20583 - for ; Sat, 23 Oct 2004 02:11:57 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id B3D75EAE4AC - for ; Sat, 23 Oct 2004 07:11:22 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 75750-09 for ; - Sat, 23 Oct 2004 06:11:32 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 560D3EADBC0 - for ; Sat, 23 Oct 2004 07:11:22 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 9D862EAE4CB - for ; Sat, 23 Oct 2004 07:08:48 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 77538-02 - for ; - Sat, 23 Oct 2004 06:08:54 +0000 (GMT) -Received: from zigo.dhs.org (as2-4-3.an.g.bonet.se [194.236.34.191]) - by svr1.postgresql.org (Postfix) with ESMTP id 29BC1EAC8F4 - for ; Sat, 23 Oct 2004 07:08:40 +0100 (BST) -Received: from zigo.zigo.dhs.org (zigo.zigo.dhs.org [192.168.0.1]) - by zigo.dhs.org (Postfix) with ESMTP - id 846788467; Sat, 23 Oct 2004 08:09:05 +0200 (CEST) -Date: Sat, 23 Oct 2004 08:09:05 +0200 (CEST) -From: Dennis Bjorklund -To: Tom Lane -cc: Robert Treat , - -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: <18876.1098478892@sss.pgh.pa.us> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=ISO-8859-1 -Content-Transfer-Encoding: 8BIT -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -On Fri, 22 Oct 2004, Tom Lane wrote: - -> behavior. The spec says you can put a numeric-GMT-offset zone in and -> get a numeric-GMT-offset zone out. We can do that and also support -> named, possibly DST-aware zones. - -So if I understand you correctly you are planning to extend the current -timestamp type to work with both named time zones and HH:MM ones? I didn't -think you wanted the last one since your plan was to store a UTC+OID where -the OID pointed to a named time zone. And I guess that you don't plan to -add 00:00, 00:01, 00:02, ... as named zones with an OID. - --- -/Dennis Björklund - - ----------------------------(end of broadcast)--------------------------- -TIP 9: the planner will ignore your desire to choose an index scan if your - joining column's datatypes do not match - -From pgsql-hackers-owner+M60329=pgman=candle.pha.pa.us@postgresql.org Mon Oct 25 12:45:39 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9PGjcf25576 - for ; Mon, 25 Oct 2004 12:45:38 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 27F433A4146 - for ; Mon, 25 Oct 2004 17:45:35 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 63030-04 for ; - Mon, 25 Oct 2004 16:45:34 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id D15153A413C - for ; Mon, 25 Oct 2004 17:45:34 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 4D4653A415D - for ; Mon, 25 Oct 2004 17:43:32 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 60516-10 - for ; - Mon, 25 Oct 2004 16:43:26 +0000 (GMT) -Received: from davinci.ethosmedia.com (server226.ethosmedia.com [209.128.84.226]) - by svr1.postgresql.org (Postfix) with ESMTP id CAB1E3A4170 - for ; Mon, 25 Oct 2004 17:43:26 +0100 (BST) -Received: from [63.195.55.98] (account josh@agliodbs.com HELO spooky) - by davinci.ethosmedia.com (CommuniGate Pro SMTP 4.1.8) - with ESMTP id 6553366 for pgsql-hackers@postgresql.org; Mon, 25 Oct 2004 09:44:52 -0700 -From: Josh Berkus -Organization: Aglio Database Solutions -To: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -Date: Mon, 25 Oct 2004 09:42:38 -0700 -User-Agent: KMail/1.6.2 -References: <20041022184636.62D39EAEE02@svr1.postgresql.org> -In-Reply-To: <20041022184636.62D39EAEE02@svr1.postgresql.org> -MIME-Version: 1.0 -Content-Disposition: inline -Content-Type: text/plain; - charset="utf-8" -Message-ID: <200410250942.38212.josh@agliodbs.com> -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Content-Transfer-Encoding: 8bit -X-MIME-Autoconverted: from quoted-printable to 8bit by candle.pha.pa.us id i9PGjcf25576 -X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on - candle.pha.pa.us -X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham - version=2.61 -Status: OR - -Tom, - -> As far as I can tell, Dennis is planning slavish adherence to the spec, -> which will mean that the datatype is unable to cope effectively with -> daylight-savings issues.  So I'm unconvinced that it will be very -> helpful to you for remembering local time in addition to true -> (universal) time. - -As somebody who codes calendar apps, I have to say that I have yet to see an -implementation of time zones which is at all useful for this purpose, -including the current implementation. My calendar apps on PostgreSQL 7.4 -use "timestamp without time zone" and keep the time zone in a seperate field. - -The reason is simple: our current implementation, which does include DST, -does not include any provision for the exceptions to DST -- such as Arizona --- or for the difference between "1 day" and "24 hours". (Try adding "30 -days" to "2004-10-05 10:00 PDT", you'll see what I mean). Nor do I see a -way out of this without raising the complexity, and configurability, level of -timezones significantly. - -So if we're going to be broken (at least from the perspective of calendar -applications) we might as well be broken in a spec-compliant way. - --- -Josh Berkus -Aglio Database Solutions -San Francisco - ----------------------------(end of broadcast)--------------------------- -TIP 6: Have you searched our list archives? - - http://archives.postgresql.org - -From pgsql-hackers-owner+M60334=pgman=candle.pha.pa.us@postgresql.org Mon Oct 25 13:56:27 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9PHuQf06230 - for ; Mon, 25 Oct 2004 13:56:26 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 024493A3C12 - for ; Mon, 25 Oct 2004 18:56:23 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 90234-01 for ; - Mon, 25 Oct 2004 17:56:21 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id B12643A3BDD - for ; Mon, 25 Oct 2004 18:56:22 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 038CC3A412E - for ; Mon, 25 Oct 2004 18:54:57 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 89206-06 - for ; - Mon, 25 Oct 2004 17:54:54 +0000 (GMT) -Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) - by svr1.postgresql.org (Postfix) with ESMTP id 0D5D13A3B06 - for ; Mon, 25 Oct 2004 18:54:54 +0100 (BST) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id i9PHsrvv021835; - Mon, 25 Oct 2004 13:54:53 -0400 (EDT) -To: Josh Berkus -cc: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: <200410250942.38212.josh@agliodbs.com> -References: <20041022184636.62D39EAEE02@svr1.postgresql.org> <200410250942.38212.josh@agliodbs.com> -Comments: In-reply-to Josh Berkus - message dated "Mon, 25 Oct 2004 09:42:38 -0700" -Date: Mon, 25 Oct 2004 13:54:53 -0400 -Message-ID: <21834.1098726893@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -Josh Berkus writes: -> The reason is simple: our current implementation, which does include DST, -> does not include any provision for the exceptions to DST -- such as Arizona - -Say what? - -regression=# set timezone to 'MST7MDT'; -SET -regression=# select now(); - now -------------------------------- - 2004-10-25 11:52:47.093538-06 -(1 row) - -regression=# set timezone to 'US/Arizona'; -SET -regression=# select now(); - now -------------------------------- - 2004-10-25 10:52:49.441559-07 -(1 row) - -> -- or for the difference between "1 day" and "24 hours". (Try adding "30 -> days" to "2004-10-05 10:00 PDT", you'll see what I mean). - -This is the point about how interval needs to treat "day" as different -from "24 hours". I agree with that; the fact that it's not done already -is just a reflection of limited supply of round tuits. I think it's -orthogonal to the question of how flexible timestamp with time zone -needs to be, though. - -> Nor do I see a way out of this without raising the complexity, and -> configurability, level of timezones significantly. - -This does not seem to me to be an argument why timestamp with time zone -ought to be incapable of dealing with DST-aware time zones. That simply -guarantees that calendar apps won't be able to use the datatype. If -they still can't use it when it can do that, then we can look at the -next blocking factor. - -> So if we're going to be broken (at least from the perspective of calendar -> applications) we might as well be broken in a spec-compliant way. - -I have not said that we can't comply with the spec. I have said that -our ambitions need to be higher than merely complying with the spec. - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate - subscribe-nomail command to majordomo@postgresql.org so that your - message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M60335=pgman=candle.pha.pa.us@postgresql.org Mon Oct 25 14:08:52 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9PI8of08458 - for ; Mon, 25 Oct 2004 14:08:51 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id EB0003A4181 - for ; Mon, 25 Oct 2004 19:08:47 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 94404-01 for ; - Mon, 25 Oct 2004 18:08:46 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 984283A417A - for ; Mon, 25 Oct 2004 19:08:47 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 058F13A3B6E - for ; Mon, 25 Oct 2004 19:06:53 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 91573-10 - for ; - Mon, 25 Oct 2004 18:06:42 +0000 (GMT) -Received: from davinci.ethosmedia.com (server226.ethosmedia.com [209.128.84.226]) - by svr1.postgresql.org (Postfix) with ESMTP id B6FB73A3AC2 - for ; Mon, 25 Oct 2004 19:06:43 +0100 (BST) -Received: from [64.81.245.111] (account josh@agliodbs.com HELO temoku.sf.agliodbs.com) - by davinci.ethosmedia.com (CommuniGate Pro SMTP 4.1.8) - with ESMTP id 6553760; Mon, 25 Oct 2004 11:08:08 -0700 -From: Josh Berkus -Reply-To: josh@agliodbs.com -Organization: Aglio Database Solutions -To: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -Date: Mon, 25 Oct 2004 11:08:52 -0700 -User-Agent: KMail/1.6.2 -cc: Tom Lane -References: <20041022184636.62D39EAEE02@svr1.postgresql.org> <200410250942.38212.josh@agliodbs.com> <21834.1098726893@sss.pgh.pa.us> -In-Reply-To: <21834.1098726893@sss.pgh.pa.us> -MIME-Version: 1.0 -Content-Disposition: inline -Content-Type: text/plain; - charset="utf-8" -Content-Transfer-Encoding: 7bit -Message-ID: <200410251108.52164.josh@agliodbs.com> -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -Tom, - -> regression=# set timezone to 'US/Arizona'; -> SET -> regression=# select now(); -> now -> ------------------------------- -> 2004-10-25 10:52:49.441559-07 - -Wow! When did that get fixed? How do I keep track of this stuff if you -guys keep fixing it? ;-) - -Of course, it would be very helpful if the result above could display -"Arizona" instead of the non-specific "-07", but I'm pretty sure that's -already a TODO. - -> This is the point about how interval needs to treat "day" as different -> from "24 hours". I agree with that; the fact that it's not done already -> is just a reflection of limited supply of round tuits. - -Well, when I first brought up the issue (2001) I was shot down on the basis of -spec-compliance, since SQL92 recognizes only Year/Month and -Day/Hour/Minute/etc. partitions. Glad it's up for consideration again. - -Come to think of it, it was Thomas Lockhart who shot down the idea of fixing -Interval, and he's retired now ... - -> This does not seem to me to be an argument why timestamp with time zone -> ought to be incapable of dealing with DST-aware time zones. That simply -> guarantees that calendar apps won't be able to use the datatype. If -> they still can't use it when it can do that, then we can look at the -> next blocking factor. - -That's definitely a progressive attitude .... pardon me for being pessimistic. - -> I have not said that we can't comply with the spec. I have said that -> our ambitions need to be higher than merely complying with the spec. - -Hmmm ... well, does the spec specifically prohibit DST, or just leave it out? - --- ---Josh - -Josh Berkus -Aglio Database Solutions -San Francisco - ----------------------------(end of broadcast)--------------------------- -TIP 7: don't forget to increase your free space map settings - -From pgsql-hackers-owner+M60336=pgman=candle.pha.pa.us@postgresql.org Mon Oct 25 14:21:12 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9PILAf10029 - for ; Mon, 25 Oct 2004 14:21:11 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 96C283A3B0E - for ; Mon, 25 Oct 2004 19:21:07 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 96954-06 for ; - Mon, 25 Oct 2004 18:21:06 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 528B73A3B06 - for ; Mon, 25 Oct 2004 19:21:07 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id E010F3A4192 - for ; Mon, 25 Oct 2004 19:19:46 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 96137-04 - for ; - Mon, 25 Oct 2004 18:19:41 +0000 (GMT) -Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) - by svr1.postgresql.org (Postfix) with ESMTP id A1BDA3A4190 - for ; Mon, 25 Oct 2004 19:19:42 +0100 (BST) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id i9PIJegg022094; - Mon, 25 Oct 2004 14:19:40 -0400 (EDT) -To: josh@agliodbs.com -cc: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: <200410251108.52164.josh@agliodbs.com> -References: <20041022184636.62D39EAEE02@svr1.postgresql.org> <200410250942.38212.josh@agliodbs.com> <21834.1098726893@sss.pgh.pa.us> <200410251108.52164.josh@agliodbs.com> -Comments: In-reply-to Josh Berkus - message dated "Mon, 25 Oct 2004 11:08:52 -0700" -Date: Mon, 25 Oct 2004 14:19:40 -0400 -Message-ID: <22093.1098728380@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -Josh Berkus writes: ->> regression=# set timezone to 'US/Arizona'; ->> SET ->> regression=# select now(); ->> now ->> ------------------------------- ->> 2004-10-25 10:52:49.441559-07 - -> Wow! When did that get fixed? How do I keep track of this stuff if you -> guys keep fixing it? ;-) - -> Of course, it would be very helpful if the result above could display -> "Arizona" instead of the non-specific "-07", but I'm pretty sure that's -> already a TODO. - -Well, that is *exactly what I'm talking about*. I want timestamp with -time zone to carry "US/Arizona" not just "-07". Obviously there needs -to be some option to get the latter displayed when that's all you want, -but internally a value of the datatype needs to be able to carry full -knowledge of which timezone it's supposed to be in. Dumbing that down -to a simple numeric GMT offset isn't good enough. - ->> I have not said that we can't comply with the spec. I have said that ->> our ambitions need to be higher than merely complying with the spec. - -> Hmmm ... well, does the spec specifically prohibit DST, or just leave it out? - -It just doesn't talk about it AFAICS. - -To comply with the spec we definitely need to be *able* to support -timezone values that are simple numeric GMT offsets. But I think we -ought also to be able to store values that are references to any of -the zic database entries. This looks to me like a straightforward -extension of the spec. - -We went to all the trouble of importing src/timezone in order that we -could make a significant upgrade in our timezone capability, and now -it's time to take the steps that that enables. Before we were limited -to the lowest-common-denominator of the libc timezone routines on all -our different platforms, but now we are not... - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 6: Have you searched our list archives? - - http://archives.postgresql.org - -From pgsql-hackers-owner+M60337=pgman=candle.pha.pa.us@postgresql.org Mon Oct 25 14:28:59 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9PISvf11118 - for ; Mon, 25 Oct 2004 14:28:58 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 928B13A3B43 - for ; Mon, 25 Oct 2004 19:28:54 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 00197-04 for ; - Mon, 25 Oct 2004 18:28:53 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 518C33A3B06 - for ; Mon, 25 Oct 2004 19:28:54 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 3AA743A3B0E - for ; Mon, 25 Oct 2004 19:27:17 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 99849-02 - for ; - Mon, 25 Oct 2004 18:27:07 +0000 (GMT) -Received: from zigo.dhs.org (as2-4-3.an.g.bonet.se [194.236.34.191]) - by svr1.postgresql.org (Postfix) with ESMTP id B62843A418A - for ; Mon, 25 Oct 2004 19:27:01 +0100 (BST) -Received: from zigo.zigo.dhs.org (zigo.zigo.dhs.org [192.168.0.1]) - by zigo.dhs.org (Postfix) with ESMTP - id 5AD8C8467; Mon, 25 Oct 2004 20:26:59 +0200 (CEST) -Date: Mon, 25 Oct 2004 20:26:59 +0200 (CEST) -From: Dennis Bjorklund -To: Josh Berkus -cc: pgsql-hackers@postgresql.org, Tom Lane -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: <200410251108.52164.josh@agliodbs.com> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=ISO-8859-1 -Content-Transfer-Encoding: 8BIT -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -On Mon, 25 Oct 2004, Josh Berkus wrote: - -> Hmmm ... well, does the spec specifically prohibit DST, or just leave it -> out? - -It doesn't discuss it. According to the spec a timestamp with time zone is -a UTC value + a HH:MM offset from GMT. And intervals in the spec is either -a year-month value or a day-time value. One can only compare year-month -values with each other and day-time values with each other. So they avoid -the problem of the how many days is a month by not allowing it. - -The spec is not a full solution, it's also not a useless solution. I'm -happy as long as the spec is a subset of what pg implements. If not then I -would like to be able to have both but with different names or something -similar (but I think that should not be needed). - --- -/Dennis Björklund - - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate - subscribe-nomail command to majordomo@postgresql.org so that your - message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M60338=pgman=candle.pha.pa.us@postgresql.org Mon Oct 25 14:54:31 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9PIsUf15052 - for ; Mon, 25 Oct 2004 14:54:30 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 3F9063A41BC - for ; Mon, 25 Oct 2004 19:54:27 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 09766-01 for ; - Mon, 25 Oct 2004 18:54:25 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id E25B63A41BA - for ; Mon, 25 Oct 2004 19:54:26 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 8FBF23A41A0 - for ; Mon, 25 Oct 2004 19:52:27 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 05475-10 - for ; - Mon, 25 Oct 2004 18:52:24 +0000 (GMT) -Received: from davinci.ethosmedia.com (server226.ethosmedia.com [209.128.84.226]) - by svr1.postgresql.org (Postfix) with ESMTP id E7C333A4182 - for ; Mon, 25 Oct 2004 19:52:24 +0100 (BST) -Received: from [64.81.245.111] (account josh@agliodbs.com HELO temoku.sf.agliodbs.com) - by davinci.ethosmedia.com (CommuniGate Pro SMTP 4.1.8) - with ESMTP id 6553982; Mon, 25 Oct 2004 11:53:49 -0700 -From: Josh Berkus -Reply-To: josh@agliodbs.com -Organization: Aglio Database Solutions -To: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -Date: Mon, 25 Oct 2004 11:54:33 -0700 -User-Agent: KMail/1.6.2 -cc: Dennis Bjorklund , Tom Lane -References: -In-Reply-To: -MIME-Version: 1.0 -Content-Disposition: inline -Content-Type: text/plain; - charset="iso-8859-1" -Content-Transfer-Encoding: 7bit -Message-ID: <200410251154.33532.josh@agliodbs.com> -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -Dennis, - -> It doesn't discuss it. According to the spec a timestamp with time zone is -> a UTC value + a HH:MM offset from GMT. And intervals in the spec is either -> a year-month value or a day-time value. One can only compare year-month -> values with each other and day-time values with each other. So they avoid -> the problem of the how many days is a month by not allowing it. - -That's not what Tom and I were talking about. The issue is that the spec -defines Days/Weeks as being an agglomeration of hours and not an atomic -entity like Months/Years are. This leads to some wierd and -calendar-breaking behavior when combined with DST, for example: - -template1=> select '2004-10-09 10:00 PDT'::TIMESTAMPTZ + '45 days'::INTERVAL -template1-> ; - ?column? ------------------------- - 2004-11-23 09:00:00-08 -(1 row) - -Because of the DST shift, you get an hour shift which is most decidely not -anything real human beings would expect from a calendar. The answer is to -try-partition INTERVAL values, as: - -Hour/Minute/Second/ms -Day/Week -Month/Year - -However, this could be considered to break the spec; certainly Thomas thought -it did. My defense is that the SQL committee made some mistakes, and -interval is a big one. - --- ---Josh - -Josh Berkus -Aglio Database Solutions -San Francisco - ----------------------------(end of broadcast)--------------------------- -TIP 2: you can get off all lists at once with the unregister command - (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) - -From pgsql-hackers-owner+M60339=pgman=candle.pha.pa.us@postgresql.org Mon Oct 25 15:07:28 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9PJ7Qf17120 - for ; Mon, 25 Oct 2004 15:07:27 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 666B73A41B0 - for ; Mon, 25 Oct 2004 20:07:23 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 13808-02 for ; - Mon, 25 Oct 2004 19:07:21 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 207F03A41AE - for ; Mon, 25 Oct 2004 20:07:23 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id D243F3A4182 - for ; Mon, 25 Oct 2004 20:04:23 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 12122-06 - for ; - Mon, 25 Oct 2004 19:04:18 +0000 (GMT) -Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) - by svr1.postgresql.org (Postfix) with ESMTP id 52FFB3A3CF2 - for ; Mon, 25 Oct 2004 20:04:18 +0100 (BST) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id i9PJ4GUm022503; - Mon, 25 Oct 2004 15:04:16 -0400 (EDT) -To: josh@agliodbs.com -cc: pgsql-hackers@postgresql.org, Dennis Bjorklund -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: <200410251154.33532.josh@agliodbs.com> -References: <200410251154.33532.josh@agliodbs.com> -Comments: In-reply-to Josh Berkus - message dated "Mon, 25 Oct 2004 11:54:33 -0700" -Date: Mon, 25 Oct 2004 15:04:16 -0400 -Message-ID: <22502.1098731056@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -Josh Berkus writes: ->> It doesn't discuss it. According to the spec a timestamp with time zone is ->> a UTC value + a HH:MM offset from GMT. And intervals in the spec is either ->> a year-month value or a day-time value. One can only compare year-month ->> values with each other and day-time values with each other. So they avoid ->> the problem of the how many days is a month by not allowing it. - -> That's not what Tom and I were talking about. The issue is that the spec -> defines Days/Weeks as being an agglomeration of hours and not an atomic -> entity like Months/Years are. - -I think though that these points are closely related. The reason the -spec does that is exactly that they are ignoring DST and so they can -assume that 1 day == 24 hours == 86400 seconds. In a DST-aware world -you have to make a separation between days and the smaller units, just -as months are separated from smaller units because there's not a fixed -conversion factor. - -To some extent the interval and timestamptz issues are orthogonal, but -I think it would be good to fix them in the same release if possible. -There will undoubtedly be some backwards-compatibility problems, and -I suppose that users would prefer to take them all at once than via -the chinese water torture method ... - -> However, this could be considered to break the spec; certainly Thomas -> thought it did. My defense is that the SQL committee made some -> mistakes, and interval is a big one. - -I'm not clear to what extent we have to actually break the spec, as -opposed to extend it, in order to do this to the "interval" type. To do -everything the spec says we need to do, we'll have to be able to make -some comparisons that aren't strictly valid (which amounts to assuming -that 1 day == 24 hours for some limited purposes) but we already do much -the same things with respect to months. (See other thread about whether -1 year == 360 days...) - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 5: Have you checked our extensive FAQ? - - http://www.postgresql.org/docs/faqs/FAQ.html - -From pgsql-hackers-owner+M60340=pgman=candle.pha.pa.us@postgresql.org Mon Oct 25 15:12:25 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9PJCNf17962 - for ; Mon, 25 Oct 2004 15:12:24 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id AA6B73A41C2 - for ; Mon, 25 Oct 2004 20:12:20 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 13059-08 for ; - Mon, 25 Oct 2004 19:12:19 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 50BAF3A41C1 - for ; Mon, 25 Oct 2004 20:12:20 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 2035E3A41B9 - for ; Mon, 25 Oct 2004 20:11:08 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 13808-08 - for ; - Mon, 25 Oct 2004 19:11:04 +0000 (GMT) -Received: from zigo.dhs.org (as2-4-3.an.g.bonet.se [194.236.34.191]) - by svr1.postgresql.org (Postfix) with ESMTP id 5D4C03A41B2 - for ; Mon, 25 Oct 2004 20:11:04 +0100 (BST) -Received: from zigo.zigo.dhs.org (zigo.zigo.dhs.org [192.168.0.1]) - by zigo.dhs.org (Postfix) with ESMTP - id 156808467; Mon, 25 Oct 2004 21:11:04 +0200 (CEST) -Date: Mon, 25 Oct 2004 21:11:04 +0200 (CEST) -From: Dennis Bjorklund -To: Josh Berkus -cc: pgsql-hackers@postgresql.org, Tom Lane -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: <200410251154.33532.josh@agliodbs.com> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=ISO-8859-1 -Content-Transfer-Encoding: 8BIT -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -On Mon, 25 Oct 2004, Josh Berkus wrote: - -> Dennis, -> -> > It doesn't discuss it. According to the spec a timestamp with time zone is -> > a UTC value + a HH:MM offset from GMT. And intervals in the spec is either -> > a year-month value or a day-time value. One can only compare year-month -> > values with each other and day-time values with each other. So they avoid -> > the problem of the how many days is a month by not allowing it. -> -> That's not what Tom and I were talking about. - -You wanted to know what the standard said, and I told what I knew. - -> The issue is that the spec defines Days/Weeks as being an agglomeration -> of hours and not an atomic entity like Months/Years are. - -I don't know what you mean with this. The standard does treat them as - -year -month -day -hour -minute -second (with fractions) - -There is no weeks there, if that is what you mean. - -> This leads to some wierd and calendar-breaking behavior when combined -> with DST, for example: -> -> template1=> select '2004-10-09 10:00 PDT'::TIMESTAMPTZ + '45 days'::INTERVAL -> template1-> ; -> ?column? -> ------------------------ -> 2004-11-23 09:00:00-08 -> (1 row) -> -> Because of the DST shift, you get an hour shift which is most decidely not -> anything real human beings would expect from a calendar. - -I don't see how the above can be caused by the representation of an -interval. The above timestamp is - -2004-10-09 10:00 PDT - -which in the standard would be - -2004-10-09 10:00 -07 - -and after the additon would be - -2004-11-23 10:00:00-07 - -Here the time zone is wrong since the standard does not know about named -zones and dst. - -An implementation like the one Tom (and I) want would start with - -2004-10-09 10:00 PDT - -and then after the addition one would get - -2004-11-23 10:00:00 PST - -At least that's my understanding of what we want and what we can get (plus -that we also need to support HH:MM tz values since those also exist in the -world, check this emails header for example). - -It's possible that you discuss something else, but that has been lost on -me so far. - --- -/Dennis Björklund - - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate - subscribe-nomail command to majordomo@postgresql.org so that your - message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M60341=pgman=candle.pha.pa.us@postgresql.org Mon Oct 25 15:15:18 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9PJFHf18332 - for ; Mon, 25 Oct 2004 15:15:17 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id D2DE53A41BD - for ; Mon, 25 Oct 2004 20:15:13 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 16799-02 for ; - Mon, 25 Oct 2004 19:15:12 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 928BF3A41A6 - for ; Mon, 25 Oct 2004 20:15:13 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 2AE4F3A41C4 - for ; Mon, 25 Oct 2004 20:13:02 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 15871-02 - for ; - Mon, 25 Oct 2004 19:12:59 +0000 (GMT) -Received: from zigo.dhs.org (as2-4-3.an.g.bonet.se [194.236.34.191]) - by svr1.postgresql.org (Postfix) with ESMTP id 6881F3A41BC - for ; Mon, 25 Oct 2004 20:12:59 +0100 (BST) -Received: from zigo.zigo.dhs.org (zigo.zigo.dhs.org [192.168.0.1]) - by zigo.dhs.org (Postfix) with ESMTP - id 09CC48467; Mon, 25 Oct 2004 21:12:59 +0200 (CEST) -Date: Mon, 25 Oct 2004 21:12:59 +0200 (CEST) -From: Dennis Bjorklund -To: Josh Berkus -cc: pgsql-hackers@postgresql.org, Tom Lane -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: <200410251154.33532.josh@agliodbs.com> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=ISO-8859-1 -Content-Transfer-Encoding: 8BIT -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -On Mon, 25 Oct 2004, Josh Berkus wrote: - -> Hour/Minute/Second/ms -> Day/Week -> Month/Year - -And just when I pressed "send" on the previous mail I got the problem -:-) - --- -/Dennis Björklund - - ----------------------------(end of broadcast)--------------------------- -TIP 2: you can get off all lists at once with the unregister command - (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) - -From pgsql-hackers-owner+M60342=pgman=candle.pha.pa.us@postgresql.org Mon Oct 25 15:20:32 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9PJKVf19055 - for ; Mon, 25 Oct 2004 15:20:32 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 0D6103A418E - for ; Mon, 25 Oct 2004 20:20:28 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 17860-02 for ; - Mon, 25 Oct 2004 19:20:26 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id AA4353A41B2 - for ; Mon, 25 Oct 2004 20:20:27 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 131A23A41BB - for ; Mon, 25 Oct 2004 20:19:02 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 16799-06 - for ; - Mon, 25 Oct 2004 19:18:51 +0000 (GMT) -Received: from zigo.dhs.org (as2-4-3.an.g.bonet.se [194.236.34.191]) - by svr1.postgresql.org (Postfix) with ESMTP id BBD663A3B06 - for ; Mon, 25 Oct 2004 20:18:52 +0100 (BST) -Received: from zigo.zigo.dhs.org (zigo.zigo.dhs.org [192.168.0.1]) - by zigo.dhs.org (Postfix) with ESMTP - id 36B248467; Mon, 25 Oct 2004 21:18:52 +0200 (CEST) -Date: Mon, 25 Oct 2004 21:18:52 +0200 (CEST) -From: Dennis Bjorklund -To: Josh Berkus -cc: pgsql-hackers@postgresql.org, Tom Lane -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: <200410251154.33532.josh@agliodbs.com> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=ISO-8859-1 -Content-Transfer-Encoding: 8BIT -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -On Mon, 25 Oct 2004, Josh Berkus wrote: - -> Hour/Minute/Second/ms -> Day/Week -> Month/Year - -This is embarrasing. I'm still a bit confused :-) - -The standard treat days as a separate entry, it does not assume that a day -is 24 hours. It restricts the hour field to the interval 0-23 so one can -never have something like 25 hours. So it does not need to worry about how -many days that translate to. - -And why do we need weeks also? - -Well, this is the last mail I send before I've been thinking about this -for a while more :-) - --- -/Dennis Björklund - - ----------------------------(end of broadcast)--------------------------- -TIP 6: Have you searched our list archives? - - http://archives.postgresql.org - -From pgsql-hackers-owner+M60344=pgman=candle.pha.pa.us@postgresql.org Mon Oct 25 15:36:30 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9PJaTf21902 - for ; Mon, 25 Oct 2004 15:36:29 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id E2F703A3CF2 - for ; Mon, 25 Oct 2004 20:36:25 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 23154-01 for ; - Mon, 25 Oct 2004 19:36:24 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 7935E3A3AC2 - for ; Mon, 25 Oct 2004 20:36:25 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 13AB03A41BC - for ; Mon, 25 Oct 2004 20:35:09 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 21223-08 - for ; - Mon, 25 Oct 2004 19:35:05 +0000 (GMT) -Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) - by svr1.postgresql.org (Postfix) with ESMTP id EA1E73A41B3 - for ; Mon, 25 Oct 2004 20:35:06 +0100 (BST) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id i9PJZ429022817; - Mon, 25 Oct 2004 15:35:04 -0400 (EDT) -To: Dennis Bjorklund -cc: Josh Berkus , pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: -References: -Comments: In-reply-to Dennis Bjorklund - message dated "Mon, 25 Oct 2004 21:18:52 +0200" -Date: Mon, 25 Oct 2004 15:35:04 -0400 -Message-ID: <22816.1098732904@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -Dennis Bjorklund writes: -> The standard treat days as a separate entry, it does not assume that a day -> is 24 hours. - -SQL92 says - - 4.5.2 Intervals - - There are two classes of intervals. One class, called year-month - intervals, has an express or implied datetime precision that in- - cludes no fields other than YEAR and MONTH, though not both are - required. The other class, called day-time intervals, has an ex- - press or implied interval precision that can include any fields - other than YEAR or MONTH. - -AFAICS the reason for this rule is that they expect all Y/M intervals to -be comparable (which they are) and they also expect all D/H/M/S intervals -to be comparable, which you can only do by assuming that 1 D == 24 H. - -It seems to me though that we can store days separately and do interval -comparisons with the assumption 1 D == 24 H, and be perfectly -SQL-compatible as far as that goes, and still make good use of the -separate day info when adding to a timestamptz that has a DST-aware -timezone. In a non-DST-aware timezone the addition will act the same as -if we weren't distinguishing days from h/m/s. Therefore, an application -using only the spec-defined features (ie, only fixed-numeric-offset -timezones) will see no deviation from the spec behavior. - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M60343=pgman=candle.pha.pa.us@postgresql.org Mon Oct 25 15:31:38 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9PJVZf21237 - for ; Mon, 25 Oct 2004 15:31:36 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 139FF3A41C0 - for ; Mon, 25 Oct 2004 20:31:32 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 21073-05 for ; - Mon, 25 Oct 2004 19:31:30 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 9E1723A41B8 - for ; Mon, 25 Oct 2004 20:31:31 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id C7A9E3A41A0 - for ; Mon, 25 Oct 2004 20:30:15 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 19517-04 - for ; - Mon, 25 Oct 2004 19:30:05 +0000 (GMT) -Received: from wolff.to (wolff.to [66.93.249.74]) - by svr1.postgresql.org (Postfix) with SMTP id 9A0483A4182 - for ; Mon, 25 Oct 2004 20:30:05 +0100 (BST) -Received: (qmail 26726 invoked by uid 500); 25 Oct 2004 19:40:57 -0000 -Date: Mon, 25 Oct 2004 14:40:57 -0500 -From: Bruno Wolff III -To: Dennis Bjorklund -cc: Josh Berkus , pgsql-hackers@postgresql.org, - Tom Lane -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -Message-ID: <20041025194057.GA26356@wolff.to> -Mail-Followup-To: Dennis Bjorklund , - Josh Berkus , pgsql-hackers@postgresql.org, - Tom Lane -References: <200410251154.33532.josh@agliodbs.com> -MIME-Version: 1.0 -Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -In-Reply-To: -User-Agent: Mutt/1.5.6i -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -On Mon, Oct 25, 2004 at 21:18:52 +0200, - Dennis Bjorklund wrote: -> On Mon, 25 Oct 2004, Josh Berkus wrote: -> -> > Hour/Minute/Second/ms -> > Day/Week -> > Month/Year -> -> This is embarrasing. I'm still a bit confused :-) -> -> The standard treat days as a separate entry, it does not assume that a day -> is 24 hours. It restricts the hour field to the interval 0-23 so one can -> never have something like 25 hours. So it does not need to worry about how -> many days that translate to. -> -> And why do we need weeks also? - -For convenience. Just like years are a group of months, weeks are a group -of days. - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M60346=pgman=candle.pha.pa.us@postgresql.org Mon Oct 25 16:10:09 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9PKA8f26656 - for ; Mon, 25 Oct 2004 16:10:08 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 07A343A4201 - for ; Mon, 25 Oct 2004 21:10:05 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 62669-07 for ; - Mon, 25 Oct 2004 20:10:03 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id ACA1C3A4200 - for ; Mon, 25 Oct 2004 21:10:04 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 251663A41FE - for ; Mon, 25 Oct 2004 21:08:18 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 62513-04 - for ; - Mon, 25 Oct 2004 20:08:15 +0000 (GMT) -Received: from zigo.dhs.org (as2-4-3.an.g.bonet.se [194.236.34.191]) - by svr1.postgresql.org (Postfix) with ESMTP id 3C94D3A3AC2 - for ; Mon, 25 Oct 2004 21:08:14 +0100 (BST) -Received: from zigo.zigo.dhs.org (zigo.zigo.dhs.org [192.168.0.1]) - by zigo.dhs.org (Postfix) with ESMTP - id 025568467; Mon, 25 Oct 2004 22:08:14 +0200 (CEST) -Date: Mon, 25 Oct 2004 22:08:14 +0200 (CEST) -From: Dennis Bjorklund -To: Tom Lane -cc: Josh Berkus , -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: <22816.1098732904@sss.pgh.pa.us> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=ISO-8859-1 -Content-Transfer-Encoding: 8BIT -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -On Mon, 25 Oct 2004, Tom Lane wrote: - -> There are two classes of intervals. One class, called year-month -> intervals, has an express or implied datetime precision that in- -> cludes no fields other than YEAR and MONTH, though not both are -> required. The other class, called day-time intervals, has an ex- -> press or implied interval precision that can include any fields -> other than YEAR or MONTH. -> -> AFAICS the reason for this rule is that they expect all Y/M intervals to -> be comparable (which they are) and they also expect all D/H/M/S intervals -> to be comparable, which you can only do by assuming that 1 D == 24 H. - -I said I was not going to send any more mails, but here we go again :-) - -The standard restrict the hour field to the interval 0-23, so there can -never be any compare between for example '1 day 1 hour' and '25 hours'. -This means that one can not add two intervals together to get a bigger -one but that it would still work to do timestamp+interval+interval. - -> It seems to me though that we can store days separately and do interval -> comparisons with the assumption 1 D == 24 H, and be perfectly -> SQL-compatible as far as that goes, and still make good use of the -> separate day info when adding to a timestamptz that has a DST-aware -> timezone. In a non-DST-aware timezone the addition will act the same as -> if we weren't distinguishing days from h/m/s. Therefore, an application -> using only the spec-defined features (ie, only fixed-numeric-offset -> timezones) will see no deviation from the spec behavior. - -I agree with this. - --- -/Dennis Björklund - - ----------------------------(end of broadcast)--------------------------- -TIP 7: don't forget to increase your free space map settings - -From pgsql-hackers-owner+M60347=pgman=candle.pha.pa.us@postgresql.org Mon Oct 25 16:20:45 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9PKKif28309 - for ; Mon, 25 Oct 2004 16:20:44 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 65B453A4215 - for ; Mon, 25 Oct 2004 21:20:40 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 66092-07 for ; - Mon, 25 Oct 2004 20:20:38 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 1DCD53A4210 - for ; Mon, 25 Oct 2004 21:20:40 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 803933A3AC2 - for ; Mon, 25 Oct 2004 21:18:00 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 64513-06 - for ; - Mon, 25 Oct 2004 20:17:49 +0000 (GMT) -Received: from davinci.ethosmedia.com (server226.ethosmedia.com [209.128.84.226]) - by svr1.postgresql.org (Postfix) with ESMTP id 726633A3CF2 - for ; Mon, 25 Oct 2004 21:17:51 +0100 (BST) -Received: from [64.81.245.111] (account josh@agliodbs.com HELO temoku.sf.agliodbs.com) - by davinci.ethosmedia.com (CommuniGate Pro SMTP 4.1.8) - with ESMTP id 6554308; Mon, 25 Oct 2004 13:19:17 -0700 -From: Josh Berkus -Reply-To: josh@agliodbs.com -Organization: Aglio Database Solutions -To: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -Date: Mon, 25 Oct 2004 13:20:01 -0700 -User-Agent: KMail/1.6.2 -cc: Dennis Bjorklund , Tom Lane -References: -In-Reply-To: -MIME-Version: 1.0 -Content-Disposition: inline -Content-Type: text/plain; - charset="iso-8859-1" -Content-Transfer-Encoding: 7bit -Message-ID: <200410251320.01311.josh@agliodbs.com> -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -Dennis, - -> An implementation like the one Tom (and I) want would start with -> -> 2004-10-09 10:00 PDT -> -> and then after the addition one would get -> -> 2004-11-23 10:00:00 PST - -Sounds like we're on the same page then. - -> The standard restrict the hour field to the interval 0-23, so there can -> never be any compare between for example '1 day 1 hour' and '25 hours'. -> This means that one can not add two intervals together to get a bigger -> one but that it would still work to do timestamp+interval+interval. - -Hour field of the timestamp, or hour field of interval? There a world of -difference. - -As long as we're willing to live with the understanding that +1day 1 hour may -produce a slightly different result than + 25 hours, I don't see the problem. -Currently I can add +900 hours if I like, postgreSQL will support it. - --- ---Josh - -Josh Berkus -Aglio Database Solutions -San Francisco - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate - subscribe-nomail command to majordomo@postgresql.org so that your - message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M60348=pgman=candle.pha.pa.us@postgresql.org Mon Oct 25 16:24:27 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9PKOQf28960 - for ; Mon, 25 Oct 2004 16:24:26 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 2AD843A419B - for ; Mon, 25 Oct 2004 21:24:22 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 68942-02 for ; - Mon, 25 Oct 2004 20:24:20 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id E08EE3A4182 - for ; Mon, 25 Oct 2004 21:24:21 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id C8C413A419B - for ; Mon, 25 Oct 2004 21:22:31 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 68322-02 - for ; - Mon, 25 Oct 2004 20:22:28 +0000 (GMT) -Received: from zigo.dhs.org (as2-4-3.an.g.bonet.se [194.236.34.191]) - by svr1.postgresql.org (Postfix) with ESMTP id BB2F53A41EC - for ; Mon, 25 Oct 2004 21:22:29 +0100 (BST) -Received: from zigo.zigo.dhs.org (zigo.zigo.dhs.org [192.168.0.1]) - by zigo.dhs.org (Postfix) with ESMTP - id 2B7368467; Mon, 25 Oct 2004 22:22:29 +0200 (CEST) -Date: Mon, 25 Oct 2004 22:22:29 +0200 (CEST) -From: Dennis Bjorklund -To: Josh Berkus -cc: pgsql-hackers@postgresql.org, Tom Lane -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: <200410251320.01311.josh@agliodbs.com> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=ISO-8859-1 -Content-Transfer-Encoding: 8BIT -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -On Mon, 25 Oct 2004, Josh Berkus wrote: - -> > The standard restrict the hour field to the interval 0-23, so there can -> > never be any compare between for example '1 day 1 hour' and '25 hours'. -> > This means that one can not add two intervals together to get a bigger -> > one but that it would still work to do timestamp+interval+interval. -> -> Hour field of the timestamp, or hour field of interval? There a world of -> difference. - -Hour field of an interval can be 0-23 according to the spec (doesn't say -that we need that restriction, but we do need to understand what the spec -say). - --- -/Dennis Björklund - - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate - subscribe-nomail command to majordomo@postgresql.org so that your - message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M60349=pgman=candle.pha.pa.us@postgresql.org Mon Oct 25 16:34:48 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9PKYlf00828 - for ; Mon, 25 Oct 2004 16:34:47 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id D7AD63A3BF2 - for ; Mon, 25 Oct 2004 21:34:43 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 72374-02 for ; - Mon, 25 Oct 2004 20:34:41 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 99E183A3B88 - for ; Mon, 25 Oct 2004 21:34:43 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 534763A3C84 - for ; Mon, 25 Oct 2004 21:32:42 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 71412-04 - for ; - Mon, 25 Oct 2004 20:32:36 +0000 (GMT) -Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) - by svr1.postgresql.org (Postfix) with ESMTP id 798CA3A3B88 - for ; Mon, 25 Oct 2004 21:32:38 +0100 (BST) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id i9PKWbDK023330; - Mon, 25 Oct 2004 16:32:37 -0400 (EDT) -To: josh@agliodbs.com -cc: pgsql-hackers@postgresql.org, Dennis Bjorklund -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: <200410251320.01311.josh@agliodbs.com> -References: <200410251320.01311.josh@agliodbs.com> -Comments: In-reply-to Josh Berkus - message dated "Mon, 25 Oct 2004 13:20:01 -0700" -Date: Mon, 25 Oct 2004 16:32:37 -0400 -Message-ID: <23329.1098736357@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -Josh Berkus writes: -> As long as we're willing to live with the understanding that +1day 1 hour may -> produce a slightly different result than + 25 hours, I don't see the problem. - -Right, which is exactly why we can't accept the spec's restriction that -the hour field be limited to 0-23. People may legitimately want to add -48 hours to a timestamp, and *not* have that mean the same as adding -"2 days". Besides, we would have a backwards-compatibility problem if -we tried to forbid it, since as you note we've always accepted such input. - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 5: Have you checked our extensive FAQ? - - http://www.postgresql.org/docs/faqs/FAQ.html - -From pgsql-hackers-owner+M60371=pgman=candle.pha.pa.us@postgresql.org Mon Oct 25 21:26:49 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9Q1Qmf13334 - for ; Mon, 25 Oct 2004 21:26:49 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id A8E2F3A42A1 - for ; Tue, 26 Oct 2004 02:26:43 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 52441-04 for ; - Tue, 26 Oct 2004 01:26:39 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 5F7503A42A0 - for ; Tue, 26 Oct 2004 02:26:43 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 02D5D3A427F - for ; Tue, 26 Oct 2004 02:25:27 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 52284-03 - for ; - Tue, 26 Oct 2004 01:25:22 +0000 (GMT) -Received: from houston.familyhealth.com.au (houston.au.fhnetwork.com [203.22.197.21]) - by svr1.postgresql.org (Postfix) with ESMTP id 5C0723A42A0 - for ; Tue, 26 Oct 2004 02:25:25 +0100 (BST) -Received: from houston.familyhealth.com.au (localhost [127.0.0.1]) - by houston.familyhealth.com.au (Postfix) with ESMTP id 1BDCB2506C; - Tue, 26 Oct 2004 09:25:25 +0800 (WST) -Received: from [192.168.0.40] (work-40.internal [192.168.0.40]) - by houston.familyhealth.com.au (Postfix) with ESMTP id 14E452506B; - Tue, 26 Oct 2004 09:25:25 +0800 (WST) -Message-ID: <417DA785.7010208@familyhealth.com.au> -Date: Tue, 26 Oct 2004 09:25:25 +0800 -From: Christopher Kings-Lynne -User-Agent: Mozilla Thunderbird 0.8 (Windows/20040913) -X-Accept-Language: en-us, en -MIME-Version: 1.0 -To: Tom Lane -cc: josh@agliodbs.com, pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -References: <20041022184636.62D39EAEE02@svr1.postgresql.org> <200410250942.38212.josh@agliodbs.com> <21834.1098726893@sss.pgh.pa.us> <200410251108.52164.josh@agliodbs.com> <22093.1098728380@sss.pgh.pa.us> -In-Reply-To: <22093.1098728380@sss.pgh.pa.us> -Content-Type: text/plain; charset=ISO-8859-1; format=flowed -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - ->>>regression=# set timezone to 'US/Arizona'; ->>>SET ->>>regression=# select now(); ->>>now ->>>------------------------------- ->>>2004-10-25 10:52:49.441559-07 -> -> ->>Wow! When did that get fixed? How do I keep track of this stuff if you ->>guys keep fixing it? ;-) - -That's worked for ages. What doesn't work is this: - -usatest=# select current_timestamp at time zone 'US/Arizona'; -ERROR: time zone "us/arizona" not recognized - -Chris - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M60372=pgman=candle.pha.pa.us@postgresql.org Mon Oct 25 21:41:27 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9Q1fPf15148 - for ; Mon, 25 Oct 2004 21:41:26 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id A941F3A4278 - for ; Tue, 26 Oct 2004 02:41:20 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 54196-09 for ; - Tue, 26 Oct 2004 01:41:16 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id 50EEA3A4255 - for ; Tue, 26 Oct 2004 02:41:20 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 34AC13A4288 - for ; Tue, 26 Oct 2004 02:39:55 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 54779-07 - for ; - Tue, 26 Oct 2004 01:39:42 +0000 (GMT) -Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) - by svr1.postgresql.org (Postfix) with ESMTP id 3E9493A1D91 - for ; Tue, 26 Oct 2004 02:39:45 +0100 (BST) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id i9Q1ditA003223; - Mon, 25 Oct 2004 21:39:46 -0400 (EDT) -To: Dennis Bjorklund -cc: Robert Treat , pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: -References: -Comments: In-reply-to Dennis Bjorklund - message dated "Sat, 23 Oct 2004 08:09:05 +0200" -Date: Mon, 25 Oct 2004 21:39:43 -0400 -Message-ID: <3222.1098754783@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -Dennis Bjorklund writes: -> So if I understand you correctly you are planning to extend the current -> timestamp type to work with both named time zones and HH:MM ones? I didn't -> think you wanted the last one since your plan was to store a UTC+OID where -> the OID pointed to a named time zone. And I guess that you don't plan to -> add 00:00, 00:01, 00:02, ... as named zones with an OID. - -I missed getting back to you on this, but I think we can do both. Some -random points: - -* Once we expand timestamptz to bigger than 8 bytes, there's essentially -zero cost to making it 12 bytes, and for that matter we could go to 16 -without much penalty, because of alignment considerations. So there's -plenty of space. - -* What we need is to be able to represent either a fixed offset from UTC -or a reference of some kind to a zic database entry. The most -bit-splurging way of doing the former is a signed offset in seconds from -Greenwich, which would take 17 bits. It'd be good enough to represent -the offset in minutes, which needs only 11 bits. - -* I suggested OIDs for referencing zic entries, but we don't have to do -that; any old mapping table will do. 16 bits would surely be plenty to -assign a unique label to every present and future zic entry. - -* My inclination therefore is to extend timestamptz with two 16-bit -fields, one being the offset from UTC (in minutes) and one being the -zic identifier. If the identifier is zero then it's a straight numeric -offset from UTC and the offset field is all you need (this is the SQL -spec compatible case). If the identifier is not zero then it gives you -an index to look up the timezone rules. However, there is no need for -the offset field to go to waste; we should store the offset anyway, -since that might save a trip to the zic database in some cases. - -* It's not clear to me yet whether the stored offset in the second case -should be the zone's standard UTC offset (thus always the same for a -given zone ID) or the current-time offset for the timestamp (thus -different if the timestamp is in daylight-savings or standard time). - -* If we store the current-time offset then it almost doesn't matter -whether the timestamp itself is stored as a UTC or local time value; -you can trivially translate either to the other by adding or subtracting -the offset (*60). But I'm inclined to store UTC for consistency with -past practice, and because it will make comparisons a bit faster: you -can compare the timestamps without adjusting first. Generally I think -comparisons ought to be the best-optimized operations in a Postgres -datatype, because index operations will do a ton of 'em. (We definitely -do NOT want to have to visit the zic database in order to compare two -timestamptz values.) - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M60373=pgman=candle.pha.pa.us@postgresql.org Mon Oct 25 21:53:52 2004 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id i9Q1rpf17038 - for ; Mon, 25 Oct 2004 21:53:51 -0400 (EDT) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 1B6153A42AC - for ; Tue, 26 Oct 2004 02:53:46 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 58642-09 for ; - Tue, 26 Oct 2004 01:53:42 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id CE6DE3A42A7 - for ; Tue, 26 Oct 2004 02:53:45 +0100 (BST) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 768EE3A429E - for ; Tue, 26 Oct 2004 02:52:13 +0100 (BST) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 58205-07 - for ; - Tue, 26 Oct 2004 01:52:07 +0000 (GMT) -Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) - by svr1.postgresql.org (Postfix) with ESMTP id 397C43A4294 - for ; Tue, 26 Oct 2004 02:52:11 +0100 (BST) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id i9Q1q07B003307; - Mon, 25 Oct 2004 21:52:00 -0400 (EDT) -To: Christopher Kings-Lynne -cc: josh@agliodbs.com, pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] timestamp with time zone a la sql99 -In-Reply-To: <417DA785.7010208@familyhealth.com.au> -References: <20041022184636.62D39EAEE02@svr1.postgresql.org> <200410250942.38212.josh@agliodbs.com> <21834.1098726893@sss.pgh.pa.us> <200410251108.52164.josh@agliodbs.com> <22093.1098728380@sss.pgh.pa.us> <417DA785.7010208@familyhealth.com.au> -Comments: In-reply-to Christopher Kings-Lynne - message dated "Tue, 26 Oct 2004 09:25:25 +0800" -Date: Mon, 25 Oct 2004 21:52:00 -0400 -Message-ID: <3306.1098755520@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: OR - -Christopher Kings-Lynne writes: -> That's worked for ages. What doesn't work is this: - -> usatest=# select current_timestamp at time zone 'US/Arizona'; -> ERROR: time zone "us/arizona" not recognized - -Right, and similarly you can do - -regression=# select '2004-10-25 21:32:33.430222 MST'::timestamptz; - timestamptz -------------------------------- - 2004-10-26 00:32:33.430222-04 -(1 row) - -but not - -regression=# select '2004-10-25 21:32:33.430222 US/Arizona'::timestamptz; -ERROR: invalid input syntax for type timestamp with time zone: "2004-10-25 21:32:33.430222 US/Arizona" - -I would like to see both of these cases working in 8.1; and furthermore -I'd like to see the timezone specs coming back as entered, not as bare -numeric offsets. (This will need to be adjustable via a DateStyle -option, of course, but I want the information to be in there whether it -is displayed or not.) - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 7: don't forget to increase your free space map settings - diff --git a/doc/TODO.detail/wal b/doc/TODO.detail/wal deleted file mode 100644 index bfdcb1613d..0000000000 --- a/doc/TODO.detail/wal +++ /dev/null @@ -1,2791 +0,0 @@ -From cjs@cynic.net Thu Jun 20 22:18:27 2002 -Return-path: -Received: from academic.cynic.net (academic.cynic.net [63.144.177.3]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5L2IPo22195 - for ; Thu, 20 Jun 2002 22:18:26 -0400 (EDT) -Received: from angelic-academic.cvpn.cynic.net (angelic-academic.cvpn.cynic.net [198.73.220.224]) - by academic.cynic.net (Postfix) with ESMTP - id 88216F821; Fri, 21 Jun 2002 02:18:17 +0000 (UTC) -Date: Fri, 21 Jun 2002 11:18:14 +0900 (JST) -From: Curt Sampson -To: Bruce Momjian -cc: Michael Loftis , mlw , - PostgreSQL-development -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: <200206210158.g5L1wFk20118@candle.pha.pa.us> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Status: RO - -On Thu, 20 Jun 2002, Bruce Momjian wrote: - -> > MS SQL Server has an interesting way of dealing with this. They have a -> > "torn" bit in each 512-byte chunk of a page, and this bit is set the -> > same for each chunk. When they are about to write out a page, they first -> > flip all of the torn bits and then do the write. If the write does not -> > complete due to a system crash or whatever, this can be detected later -> > because the torn bits won't match across the entire page. -> -> I was wondering, how does knowing the block is corrupt help MS SQL? - -I'm trying to recall, but I can't off hand. I'll have to look it -up in my Inside SQL Server book, which is at home right now, -unfortunately. I'll bring the book into work and let you know the -details later. - -> Right now, we write changed pages to WAL, then later write them to disk. - -Ah. You write the entire page? MS writes only the changed tuple. -And DB2, in fact, goes one better and writes only the part of the -tuple up to the change, IIRC. Thus, if you put smaller and/or more -frequently changed columns first, you'll have smaller logs. - -> I have always been looking for a way to prevent these WAL writes. The -> 512-byte bit seems interesting, but how does it help? - -Well, this would at least let you reduce the write to the 512-byte -chunk that changed, rather than writing the entire 8K page. - -> And how does the bit help them with partial block writes? Is the bit at -> the end of the block? Is that reliable? - -The bit is somewhere within every 512 byte "disk page" within the -8192 byte "filesystem/database page." So an 8KB page is divided up -like this: - - | <----------------------- 8 Kb ----------------------> | - - | 512b | 512b | 512b | 512b | 512b | 512b | 512b | 512b | - -Thus, the tear bits start out like this: - - | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | - -After a successful write of the entire page, you have this: - - | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | - -If the write is unsuccessful, you end up with something like this: - - | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | - -And now you know which parts of your page got written, and which -parts didn't. - -cjs --- -Curt Sampson +81 90 7737 2974 http://www.netbsd.org - Don't you know, in this new Dark Age, we're all light. --XTC - - -From cjs@cynic.net Sat Jun 22 04:41:54 2002 -Return-path: -Received: from academic.cynic.net ([63.144.177.3]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5M8fpF04711 - for ; Sat, 22 Jun 2002 04:41:53 -0400 (EDT) -Received: from angelic-academic.cvpn.cynic.net (angelic-academic.cvpn.cynic.net [198.73.220.224]) - by academic.cynic.net (Postfix) with ESMTP - id 415C8F820; Sat, 22 Jun 2002 08:41:33 +0000 (UTC) -Date: Sat, 22 Jun 2002 17:41:30 +0900 (JST) -From: Curt Sampson -To: Tom Lane -cc: Bruce Momjian , Michael Loftis , - mlw , - PostgreSQL-development -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: <19332.1024668861@sss.pgh.pa.us> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Status: RO - -On Fri, 21 Jun 2002, Tom Lane wrote: - -> Curt Sampson writes: -> > And now you know which parts of your page got written, and which -> > parts didn't. -> -> Yes ... and what do you *do* about it? - -Ok. Here's the extract from _Inside Microsoft SQL Server 7.0_, page 207: - - torn page detection When TRUE, this option causes a bit to be - flipped for each 512-byte sector in a database page (8 KB) - whenever the page is written to disk. This option allows - SQL Server to detect incomplete I/O operations caused by - power failures or other system outages. If a bit is in the - wrong state when the page is later read by SQL Server, this - means the page was written incorrectly; a torn page has - been detected. Although SQL Server database pages are 8 - KB, disks perform I/O operations using 512-byte sectors. - Therefore, 16 sectors are written per database page. A - torn page can occur if the system crashes (for example, - because of power failure) between the time the operating - system writes the first 512-byte sector to disk and the - completion of the 8-KB I/O operation. If the first sector - of a database page is successfully written before the crash, - it will appear that the database page on disk was updated, - although it might not have succeeded. Using battery-backed - disk caches can ensure that data is [sic] successfully - written to disk or not written at all. In this case, don't - set torn page detection to TRUE, as it isn't needed. If a - torn page is detected, the database will need to be restored - from backup because it will be physically inconsistent. - -As I understand it, this is not a problem for postgres becuase the -entire page is written to the log. So postgres is safe, but quite -inefficient. (It would be much more efficient to write just the -changed tuple, or even just the changed values within the tuple, -to the log.) - -Adding these torn bits would allow posgres at least to write to -the log just the 512-byte sectors that have changed, rather than -the entire 8 KB page. - -cjs --- -Curt Sampson +81 90 7737 2974 http://www.netbsd.org - Don't you know, in this new Dark Age, we're all light. --XTC - - -From pgsql-hackers-owner+M24060@postgresql.org Sat Jun 22 18:31:21 2002 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5MMVKF20014 - for ; Sat, 22 Jun 2002 18:31:20 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP - id 0ADFE476090; Sat, 22 Jun 2002 18:31:10 -0400 (EDT) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 6B372475A96; Sat, 22 Jun 2002 18:28:42 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP id 47AD2475935 - for ; Sat, 22 Jun 2002 18:28:40 -0400 (EDT) -Received: from hades.usol.com (hades.usol.com [208.232.58.41]) - by postgresql.org (Postfix) with ESMTP id 1D5DA476166 - for ; Sat, 22 Jun 2002 18:23:16 -0400 (EDT) -Received: from 01-081.024.popsite.net (01-081.024.popsite.net [216.126.160.81]) - by hades.usol.com (8.11.6/8.11.6) with ESMTP id g5MMMOj11344; - Sat, 22 Jun 2002 18:22:25 -0400 -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -From: "J. R. Nield" -To: Bruce Momjian -cc: Curt Sampson , Michael Loftis , - mlw , - PostgreSQL Hacker , - Tom Lane -In-Reply-To: <200206210158.g5L1wFk20118@candle.pha.pa.us> -References: <200206210158.g5L1wFk20118@candle.pha.pa.us> -Content-Type: text/plain -Content-Transfer-Encoding: 7bit -Message-ID: <1024784514.1793.242.camel@localhost.localdomain> -MIME-Version: 1.0 -X-Mailer: Ximian Evolution 1.0.3 (1.0.3-6) -Date: 22 Jun 2002 18:22:58 -0400 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: ROr - -On Thu, 2002-06-20 at 21:58, Bruce Momjian wrote: -> I was wondering, how does knowing the block is corrupt help MS SQL? -> Right now, we write changed pages to WAL, then later write them to disk. -> I have always been looking for a way to prevent these WAL writes. The -> 512-byte bit seems interesting, but how does it help? -> -> And how does the bit help them with partial block writes? Is the bit at -> the end of the block? Is that reliable? -> - -My understanding of this is as follows: - -1) On most commercial systems, if you get a corrupted block (from -partial write or whatever) you need to restore the file(s) from the most -recent backup, and replay the log from the log archive (usually only the -damaged files will be written to during replay). - -2) If you can't deal with the downtime to recover the file, then EMC, -Sun, or IBM will sell you an expensive disk array with an NVRAM cache -that will do atomic writes. Some plain-vanilla SCSI disks are also -capable of atomic writes, though usually they don't use NVRAM to do it. - -The database must then make sure that each page-write gets translated -into exactly one SCSI-level write. This is one reason why ORACLE and -Sybase recommend that you use raw disk partitions for high availability. -Some operating systems support this through the filesystem, but it is OS -dependent. I think Solaris 7 & 8 has support for this, but I'm not sure. - -PostgreSQL has trouble because it can neither archive logs for replay, -nor use raw disk partitions. - - -One other point: - -Page pre-image logging is fundamentally the same as what Jim Grey's -book[1] would call "careful writes". I don't believe they should be in -the XLOG, because we never need to keep the pre-images after we're sure -the buffer has made it to the disk. Instead, we should have the buffer -IO routines implement ping-pong writes of some kind if we want -protection from partial writes. - - -Does any of this make sense? - - - -;jrnield - - -[1] Grey, J. and Reuter, A. (1993). "Transaction Processing: Concepts - and Techniques". Morgan Kaufmann. - --- -J. R. Nield -jrnield@usol.com - - - - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M24068@postgresql.org Sun Jun 23 08:40:27 2002 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5NCeQF01601 - for ; Sun, 23 Jun 2002 08:40:27 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP - id 8AC4B475CBC; Sun, 23 Jun 2002 08:40:22 -0400 (EDT) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 4683647599D; Sun, 23 Jun 2002 08:37:40 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP id 0D57847592A - for ; Sun, 23 Jun 2002 08:37:38 -0400 (EDT) -Received: from hades.usol.com (hades.usol.com [208.232.58.41]) - by postgresql.org (Postfix) with ESMTP id 75326475876 - for ; Sun, 23 Jun 2002 08:37:36 -0400 (EDT) -Received: from 08-032.024.popsite.net (08-032.024.popsite.net [66.19.4.32]) - by hades.usol.com (8.11.6/8.11.6) with ESMTP id g5NCbNj02111; - Sun, 23 Jun 2002 08:37:23 -0400 -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -From: "J. R. Nield" -To: Bruce Momjian -cc: Curt Sampson , Michael Loftis , - mlw , - PostgreSQL Hacker , - Tom Lane -In-Reply-To: <200206222317.g5MNHBn23427@candle.pha.pa.us> -References: <200206222317.g5MNHBn23427@candle.pha.pa.us> -Content-Type: text/plain -Content-Transfer-Encoding: 7bit -X-Mailer: Ximian Evolution 1.0.3 (1.0.3-6) -Date: 23 Jun 2002 08:37:53 -0400 -Message-ID: <1024835880.1793.264.camel@localhost.localdomain> -MIME-Version: 1.0 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: RO - -On Sat, 2002-06-22 at 19:17, Bruce Momjian wrote: -> J. R. Nield wrote: -> > One other point: -> > -> > Page pre-image logging is fundamentally the same as what Jim Grey's -> > book[1] would call "careful writes". I don't believe they should be in -> > the XLOG, because we never need to keep the pre-images after we're sure -> > the buffer has made it to the disk. Instead, we should have the buffer -> > IO routines implement ping-pong writes of some kind if we want -> > protection from partial writes. -> -> Ping-pong writes to where? We have to fsync, and rather than fsync that -> area and WAL, we just do WAL. Not sure about a win there. -> - -The key question is: do we have some method to ensure that the OS -doesn't do the writes in parallel? - -If the OS will ensure that one of the two block writes of a ping-pong -completes before the other starts, then we don't need to fsync() at -all. - -The only thing we are protecting against is the possibility of both -writes being partial. If neither is done, that's fine because WAL will -protect us. If the first write is partial, we will detect that and use -the old data from the other, then recover from WAL. If the first is -complete but the second is partial, then we detect that and use the -newer block from the first write. If the second is complete but the -first is partial, we detect that and use the newer block from the second -write. - -So does anyone know a way to prevent parallel writes in one of the -common unix standards? Do they say anything about this? - -It would seem to me that if the same process does both ping-pong writes, -then there should be a cheap way to enforce a serial order. I could be -wrong though. - -As to where the first block of the ping-pong should go, maybe we could -reserve a file with nBlocks space for them, and write the information -about which block was being written to the XLOG for use in recovery. -There are many other ways to do it. - -;jrnield - --- -J. R. Nield -jrnield@usol.com - - - - ----------------------------(end of broadcast)--------------------------- -TIP 6: Have you searched our list archives? - -http://archives.postgresql.org - -From cjs@cynic.net Sun Jun 23 09:33:29 2002 -Return-path: -Received: from academic.cynic.net (academic.cynic.net [63.144.177.3]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5NDXSF11543 - for ; Sun, 23 Jun 2002 09:33:28 -0400 (EDT) -Received: from angelic-academic.cvpn.cynic.net (angelic-academic.cvpn.cynic.net [198.73.220.224]) - by academic.cynic.net (Postfix) with ESMTP - id A83ABF820; Sun, 23 Jun 2002 13:33:15 +0000 (UTC) -Date: Sun, 23 Jun 2002 22:33:07 +0900 (JST) -From: Curt Sampson -To: "J. R. Nield" -cc: Bruce Momjian , Michael Loftis , - mlw , - PostgreSQL Hacker , - Tom Lane -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: <1024835880.1793.264.camel@localhost.localdomain> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Status: RO - -On 23 Jun 2002, J. R. Nield wrote: - -> On Sat, 2002-06-22 at 19:17, Bruce Momjian wrote: -> > J. R. Nield wrote: -> > > One other point: -> > > -> > > Page pre-image logging is fundamentally the same as what Jim Grey's -> > > book[1] would call "careful writes". I don't believe they should be in -> > > the XLOG, because we never need to keep the pre-images after we're sure -> > > the buffer has made it to the disk. Instead, we should have the buffer -> > > IO routines implement ping-pong writes of some kind if we want -> > > protection from partial writes. -> > -> > Ping-pong writes to where? We have to fsync, and rather than fsync that -> > area and WAL, we just do WAL. Not sure about a win there. - -Presumably the win is that, "we never need to keep the pre-images -after we're sure the buffer has made it to the disk." So the -pre-image log can be completely ditched when we shut down the -server, so a full system sync, or whatever. This keeps the log file -size down, which means faster recovery, less to back up (when we -start getting transaction logs that can be backed up), etc. - -This should also allow us to disable completely the ping-pong writes -if we have a disk subsystem that we trust. (E.g., a disk array with -battery backed memory.) That would, in theory, produce a nice little -performance increase when lots of inserts and/or updates are being -committed, as we have much, much less to write to the log file. - -Are there stats that track, e.g., the bandwidth of writes to the -log file? I'd be interested in knowing just what kind of savings -one might see by doing this. - -> The key question is: do we have some method to ensure that the OS -> doesn't do the writes in parallel?... -> It would seem to me that if the same process does both ping-pong writes, -> then there should be a cheap way to enforce a serial order. I could be -> wrong though. - -Well, whether or not there's a cheap way depends on whether you consider -fsync to be cheap. :-) - -cjs --- -Curt Sampson +81 90 7737 2974 http://www.netbsd.org - Don't you know, in this new Dark Age, we're all light. --XTC - - -From pgsql-hackers-owner+M24073@postgresql.org Sun Jun 23 11:19:59 2002 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5NFJxF19785 - for ; Sun, 23 Jun 2002 11:19:59 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP - id 0BD5B475E79; Sun, 23 Jun 2002 11:19:55 -0400 (EDT) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 5C0CB475D6A; Sun, 23 Jun 2002 11:19:50 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP id E2353475C4B - for ; Sun, 23 Jun 2002 11:19:47 -0400 (EDT) -Received: from sss.pgh.pa.us (unknown [192.204.191.242]) - by postgresql.org (Postfix) with ESMTP id 746F8475AEA - for ; Sun, 23 Jun 2002 11:19:46 -0400 (EDT) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id g5NFJF108464; - Sun, 23 Jun 2002 11:19:15 -0400 (EDT) -To: Curt Sampson -cc: "J. R. Nield" , Bruce Momjian , - Michael Loftis , mlw , - PostgreSQL Hacker -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: -References: -Comments: In-reply-to Curt Sampson - message dated "Sun, 23 Jun 2002 22:33:07 +0900" -Date: Sun, 23 Jun 2002 11:19:15 -0400 -Message-ID: <8461.1024845555@sss.pgh.pa.us> -From: Tom Lane -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: RO - -Curt Sampson writes: -> This should also allow us to disable completely the ping-pong writes -> if we have a disk subsystem that we trust. - -If we have a disk subsystem we trust, we just disable fsync on the -WAL and the performance issue largely goes away. - -I concur with Bruce: the reason we keep page images in WAL is to -minimize the number of places we have to fsync, and thus the amount of -head movement required for a commit. Putting the page images elsewhere -cannot be a win AFAICS. - -> Well, whether or not there's a cheap way depends on whether you consider -> fsync to be cheap. :-) - -It's never cheap :-( - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 5: Have you checked our extensive FAQ? - -http://www.postgresql.org/users-lounge/docs/faq.html - -From cjs@cynic.net Sun Jun 23 12:10:44 2002 -Return-path: -Received: from academic.cynic.net (academic.cynic.net [63.144.177.3]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5NGAgF22907 - for ; Sun, 23 Jun 2002 12:10:43 -0400 (EDT) -Received: from angelic-academic.cvpn.cynic.net (angelic-academic.cvpn.cynic.net [198.73.220.224]) - by academic.cynic.net (Postfix) with ESMTP - id 57BFDF820; Sun, 23 Jun 2002 16:10:35 +0000 (UTC) -Date: Mon, 24 Jun 2002 01:10:26 +0900 (JST) -From: Curt Sampson -To: Tom Lane -cc: "J. R. Nield" , Bruce Momjian , - Michael Loftis , mlw , - PostgreSQL Hacker -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: <8461.1024845555@sss.pgh.pa.us> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Status: RO - -On Sun, 23 Jun 2002, Tom Lane wrote: - -> Curt Sampson writes: -> > This should also allow us to disable completely the ping-pong writes -> > if we have a disk subsystem that we trust. -> -> If we have a disk subsystem we trust, we just disable fsync on the -> WAL and the performance issue largely goes away. - -No, you can't do this. If you don't fsync(), there's no guarantee -that the write ever got out of the computer's buffer cache and to -the disk subsystem in the first place. - -> I concur with Bruce: the reason we keep page images in WAL is to -> minimize the number of places we have to fsync, and thus the amount of -> head movement required for a commit. - -An fsync() does not necessarially cause head movement, or any real -disk writes at all. If you're writing to many external disk arrays, -for example, the fsync() ensures that the data are in the disk array's -non-volatile or UPS-backed RAM, no more. The array might hold the data -for quite some time before it actually writes it to disk. - -But you're right that it's faster, if you're going to write out changed -pages and have have the ping-pong file and the transaction log on the -same disk, just to write out the entire page to the transaction log. - -So what we would really need to implement, if we wanted to be more -efficient with trusted disk subsystems, would be the option of writing -to the log only the changed row or changed part of the row, or writing -the entire changed page. I don't know how hard this would be.... - -> > Well, whether or not there's a cheap way depends on whether you consider -> > fsync to be cheap. :-) -> -> It's never cheap :-( - -Actually, with a good external RAID system with non-volatile RAM, -it's a good two to four orders of magnitude cheaper than writing to a -directly connected disk that doesn't claim the write is complete until -it's physically on disk. I'd say that it qualifies as at least "not -expensive." Not that you want to do it more often than you have to -anyway.... - -cjs --- -Curt Sampson +81 90 7737 2974 http://www.netbsd.org - Don't you know, in this new Dark Age, we're all light. --XTC - - -From jrnield@usol.com Sun Jun 23 13:56:59 2002 -Return-path: -Received: from hades.usol.com (IDENT:root@hades.usol.com [208.232.58.41]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5NHusF00335 - for ; Sun, 23 Jun 2002 13:56:58 -0400 (EDT) -Received: from 04-077.024.popsite.net (04-077.024.popsite.net [216.126.163.77]) - by hades.usol.com (8.11.6/8.11.6) with ESMTP id g5NHunj18549; - Sun, 23 Jun 2002 13:56:49 -0400 -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -From: "J. R. Nield" -To: Tom Lane -cc: Curt Sampson , Bruce Momjian , - Michael Loftis , mlw , - PostgreSQL Hacker - -In-Reply-To: <8461.1024845555@sss.pgh.pa.us> -References: - <8461.1024845555@sss.pgh.pa.us> -Content-Type: text/plain -Content-Transfer-Encoding: 7bit -X-Mailer: Ximian Evolution 1.0.3 (1.0.3-6) -Date: 23 Jun 2002 13:57:19 -0400 -Message-ID: <1024855044.1793.414.camel@localhost.localdomain> -MIME-Version: 1.0 -Status: ROr - -On Sun, 2002-06-23 at 11:19, Tom Lane wrote: -> Curt Sampson writes: -> > This should also allow us to disable completely the ping-pong writes -> > if we have a disk subsystem that we trust. -> -> If we have a disk subsystem we trust, we just disable fsync on the -> WAL and the performance issue largely goes away. - -It wouldn't work because the OS buffering interferes, and we need those -WAL records on disk up to the greatest LSN of the Buffer we will be writing. - - -We already buffer WAL ourselves. We also already buffer regular pages. -Whenever we write a Buffer out of the buffer cache, it is because we -really want that page on disk and wanted to start an IO. If thats not -the case, then we should have more block buffers! - -So since we have all this buffering designed especially to meet our -needs, and since the OS buffering is in the way, can someone explain to -me why postgresql would ever open a file without the O_DSYNC flag if the -platform supports it? - - - -> -> I concur with Bruce: the reason we keep page images in WAL is to -> minimize the number of places we have to fsync, and thus the amount of -> head movement required for a commit. Putting the page images elsewhere -> cannot be a win AFAICS. - - -Why not put all the page images in a single pre-allocated file and treat -it as a ring? How could this be any worse than flushing them in the WAL -log? - -Maybe fsync would be slower with two files, but I don't see how -fdatasync would be, and most platforms support that. - -What would improve performance would be to have a dbflush process that -would work in the background flushing buffers in groups and trying to -stay ahead of ReadBuffer requests. That would let you do the temporary -side of the ping-pong as a huge O_DSYNC writev(2) request (or -fdatasync() once) and then write out the other buffers. It would also -tend to prevent the other backends from blocking on write requests. - -A dbflush could also support aio_read/aio_write on platforms like -Solaris and WindowsNT that support it. - -Am I correct that right now, buffers only get written when they get -removed from the free list for reuse? So a released dirty buffer will -sit in the buffer free list until it becomes the Least Recently Used -buffer, and will then cause a backend to block for IO in a call to -BufferAlloc? - -This would explain why we like using the OS buffer cache, and why our -performance is troublesome when we have to do synchronous IO writes, and -why fsync() takes so long to complete. All of the backends block for -each call to BufferAlloc() after a large table update by a single -backend, and then the OS buffers are always full of our "written" data. - -Am I reading the bufmgr code correctly? I already found an imaginary -race condition there once :-) - -;jnield - - -> -> > Well, whether or not there's a cheap way depends on whether you consider -> > fsync to be cheap. :-) -> -> It's never cheap :-( -> --- -J. R. Nield -jrnield@usol.com - - -From cjs@cynic.net Sun Jun 23 14:15:15 2002 -Return-path: -Received: from academic.cynic.net (academic.cynic.net [63.144.177.3]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5NIFEF01698 - for ; Sun, 23 Jun 2002 14:15:15 -0400 (EDT) -Received: from angelic-academic.cvpn.cynic.net (angelic-academic.cvpn.cynic.net [198.73.220.224]) - by academic.cynic.net (Postfix) with ESMTP - id 796E6F820; Sun, 23 Jun 2002 18:15:08 +0000 (UTC) -Date: Mon, 24 Jun 2002 03:15:01 +0900 (JST) -From: Curt Sampson -To: "J. R. Nield" -cc: Tom Lane , Bruce Momjian , - Michael Loftis , mlw , - PostgreSQL Hacker -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: <1024855044.1793.414.camel@localhost.localdomain> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Status: ROr - -On 23 Jun 2002, J. R. Nield wrote: - -> So since we have all this buffering designed especially to meet our -> needs, and since the OS buffering is in the way, can someone explain to -> me why postgresql would ever open a file without the O_DSYNC flag if the -> platform supports it? - -It's more code, if there are platforms out there that don't support -O_DYSNC. (We still have to keep the old fsync code.) On the other hand, -O_DSYNC could save us a disk arm movement over fsync() because it -appears to me that fsync is also going to force a metadata update, which -means that the inode blocks have to be written as well. - -> Maybe fsync would be slower with two files, but I don't see how -> fdatasync would be, and most platforms support that. - -Because, if both files are on the same disk, you still have to move -the disk arm from the cylinder at the current log file write point -to the cylinder at the current ping-pong file write point. And then back -again to the log file write point cylinder. - -In the end, having a ping-pong file as well seems to me unnecessary -complexity, especially when anyone interested in really good -performance is going to buy a disk subsystem that guarantees no -torn pages and thus will want to turn off the ping-pong file writes -entirely, anyway. - -cjs --- -Curt Sampson +81 90 7737 2974 http://www.netbsd.org - Don't you know, in this new Dark Age, we're all light. --XTC - - -From jrnield@usol.com Sun Jun 23 14:14:51 2002 -Return-path: -Received: from hades.usol.com (IDENT:root@hades.usol.com [208.232.58.41]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5NIEnF01649 - for ; Sun, 23 Jun 2002 14:14:50 -0400 (EDT) -Received: from 04-077.024.popsite.net (04-077.024.popsite.net [216.126.163.77]) - by hades.usol.com (8.11.6/8.11.6) with ESMTP id g5NIEkj19287; - Sun, 23 Jun 2002 14:14:46 -0400 -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -From: "J. R. Nield" -To: Curt Sampson -cc: Tom Lane , Bruce Momjian , - Michael Loftis , mlw , - PostgreSQL Hacker - -In-Reply-To: -References: -Content-Type: text/plain -Content-Transfer-Encoding: 7bit -X-Mailer: Ximian Evolution 1.0.3 (1.0.3-6) -Date: 23 Jun 2002 14:15:17 -0400 -Message-ID: <1024856120.3054.418.camel@localhost.localdomain> -MIME-Version: 1.0 -Status: RO - -On Sun, 2002-06-23 at 12:10, Curt Sampson wrote: -> -> So what we would really need to implement, if we wanted to be more -> efficient with trusted disk subsystems, would be the option of writing -> to the log only the changed row or changed part of the row, or writing -> the entire changed page. I don't know how hard this would be.... -> -We already log that stuff. The page images are in addition to the -"Logical Changes", so we could just stop logging the page images. - --- -J. R. Nield -jrnield@usol.com - - - - -From pgsql-hackers-owner+M24100@postgresql.org Mon Jun 24 13:13:41 2002 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5OHDeF08564 - for ; Mon, 24 Jun 2002 13:13:40 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP - id 05602475CBE; Mon, 24 Jun 2002 13:11:10 -0400 (EDT) -Mailbox-Line: From pgman@candle.pha.pa.us Mon Jun 24 13:11:10 2002 -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 929A247633B; Mon, 24 Jun 2002 09:26:54 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP id 962C147631A - for ; Mon, 24 Jun 2002 08:31:43 -0400 (EDT) -Mailbox-Line: From pgman@candle.pha.pa.us Mon Jun 24 08:31:43 2002 -Received: from candle.pha.pa.us (216-55-132-35.dsl.san-diego.abac.net [216.55.132.35]) - by postgresql.org (Postfix) with ESMTP id C112D475C3C - for ; Sun, 23 Jun 2002 15:35:20 -0400 (EDT) -Received: (from pgman@localhost) - by candle.pha.pa.us (8.11.6/8.10.1) id g5NJYtL07449; - Sun, 23 Jun 2002 15:34:55 -0400 (EDT) -From: Bruce Momjian -Message-ID: <200206231934.g5NJYtL07449@candle.pha.pa.us> -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: <1024855044.1793.414.camel@localhost.localdomain> -To: "J. R. Nield" -Date: Sun, 23 Jun 2002 15:34:55 -0400 (EDT) -cc: Tom Lane , Curt Sampson , - Michael Loftis , mlw , - PostgreSQL Hacker -X-Mailer: ELM [version 2.4ME+ PL97 (25)] -MIME-Version: 1.0 -Content-Transfer-Encoding: 7bit -Content-Type: text/plain; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Spam-Status: No, hits=-3.4 required=5.0 - tests=IN_REP_TO - version=2.30 -Status: RO - -J. R. Nield wrote: -> So since we have all this buffering designed especially to meet our -> needs, and since the OS buffering is in the way, can someone explain to -> me why postgresql would ever open a file without the O_DSYNC flag if the -> platform supports it? - -We sync only WAL, not the other pages, except for the sync() call we do -during checkpoint when we discard old WAL files. - -> > I concur with Bruce: the reason we keep page images in WAL is to -> > minimize the number of places we have to fsync, and thus the amount of -> > head movement required for a commit. Putting the page images elsewhere -> > cannot be a win AFAICS. -> -> -> Why not put all the page images in a single pre-allocated file and treat -> it as a ring? How could this be any worse than flushing them in the WAL -> log? -> -> Maybe fsync would be slower with two files, but I don't see how -> fdatasync would be, and most platforms support that. - -We have fdatasync option for WAL in postgresql.conf. - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 853-3000 - + If your life is a hard drive, | 830 Blythe Avenue - + Christ can be your backup. | Drexel Hill, Pennsylvania 19026 - - - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - - - -From pgsql-hackers-owner+M24091@postgresql.org Mon Jun 24 12:54:22 2002 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5OGsMF07208 - for ; Mon, 24 Jun 2002 12:54:22 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP - id 7DB7947679D; Mon, 24 Jun 2002 09:48:51 -0400 (EDT) -Mailbox-Line: From pgman@candle.pha.pa.us Mon Jun 24 09:48:51 2002 -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 3FD37476491; Mon, 24 Jun 2002 08:55:34 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP id 2769E4762E3 - for ; Mon, 24 Jun 2002 08:27:39 -0400 (EDT) -Mailbox-Line: From pgman@candle.pha.pa.us Mon Jun 24 08:27:39 2002 -Received: from candle.pha.pa.us (216-55-132-35.dsl.san-diego.abac.net [216.55.132.35]) - by postgresql.org (Postfix) with ESMTP id ED459475C61 - for ; Sun, 23 Jun 2002 15:37:08 -0400 (EDT) -Received: (from pgman@localhost) - by candle.pha.pa.us (8.11.6/8.10.1) id g5NJasa07642; - Sun, 23 Jun 2002 15:36:54 -0400 (EDT) -From: Bruce Momjian -Message-ID: <200206231936.g5NJasa07642@candle.pha.pa.us> -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: -To: Curt Sampson -Date: Sun, 23 Jun 2002 15:36:54 -0400 (EDT) -cc: "J. R. Nield" , Tom Lane , - Michael Loftis , mlw , - PostgreSQL Hacker -X-Mailer: ELM [version 2.4ME+ PL97 (25)] -MIME-Version: 1.0 -Content-Transfer-Encoding: 7bit -Content-Type: text/plain; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Spam-Status: No, hits=-3.4 required=5.0 - tests=IN_REP_TO - version=2.30 -Status: RO - -Curt Sampson wrote: -> On 23 Jun 2002, J. R. Nield wrote: -> -> > So since we have all this buffering designed especially to meet our -> > needs, and since the OS buffering is in the way, can someone explain to -> > me why postgresql would ever open a file without the O_DSYNC flag if the -> > platform supports it? -> -> It's more code, if there are platforms out there that don't support -> O_DYSNC. (We still have to keep the old fsync code.) On the other hand, -> O_DSYNC could save us a disk arm movement over fsync() because it -> appears to me that fsync is also going to force a metadata update, which -> means that the inode blocks have to be written as well. - -Again, see postgresql.conf: - -#wal_sync_method = fsync # the default varies across platforms: -# # fsync, fdatasync, open_sync, or open_datasync - -> -> > Maybe fsync would be slower with two files, but I don't see how -> > fdatasync would be, and most platforms support that. -> -> Because, if both files are on the same disk, you still have to move -> the disk arm from the cylinder at the current log file write point -> to the cylinder at the current ping-pong file write point. And then back -> again to the log file write point cylinder. -> -> In the end, having a ping-pong file as well seems to me unnecessary -> complexity, especially when anyone interested in really good -> performance is going to buy a disk subsystem that guarantees no -> torn pages and thus will want to turn off the ping-pong file writes -> entirely, anyway. - -Yes, I don't see writing to two files vs. one to be any win, especially -when we need to fsync both of them. What I would really like is to -avoid the double I/O of writing to WAL and to the data file; improving -that would be a huge win. - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 853-3000 - + If your life is a hard drive, | 830 Blythe Avenue - + Christ can be your backup. | Drexel Hill, Pennsylvania 19026 - - - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - - - -From cjs@cynic.net Sun Jun 23 20:09:44 2002 -Return-path: -Received: from academic.cynic.net (academic.cynic.net [63.144.177.3]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5O09hF00630 - for ; Sun, 23 Jun 2002 20:09:43 -0400 (EDT) -Received: from angelic-academic.cvpn.cynic.net (angelic-academic.cvpn.cynic.net [198.73.220.224]) - by academic.cynic.net (Postfix) with ESMTP - id 6F45AF820; Mon, 24 Jun 2002 00:09:38 +0000 (UTC) -Date: Mon, 24 Jun 2002 09:09:30 +0900 (JST) -From: Curt Sampson -To: Bruce Momjian -cc: "J. R. Nield" , Tom Lane , - Michael Loftis , mlw , - PostgreSQL Hacker -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: <200206231936.g5NJasa07642@candle.pha.pa.us> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Status: RO - -On Sun, 23 Jun 2002, Bruce Momjian wrote: - -> Yes, I don't see writing to two files vs. one to be any win, especially -> when we need to fsync both of them. What I would really like is to -> avoid the double I/O of writing to WAL and to the data file; improving -> that would be a huge win. - -You mean, the double I/O of writing the block to the WAL and data file? -(We'd still have to write the changed columns or whatever to the WAL, -right?) - -I'd just add an option to turn it off. If you need it, you need it; -there's no way around that except to buy hardware that is really going -to guarantee your writes (which then means you don't need it). - -cjs --- -Curt Sampson +81 90 7737 2974 http://www.netbsd.org - Don't you know, in this new Dark Age, we're all light. --XTC - - -From jrnield@usol.com Sun Jun 23 21:28:58 2002 -Return-path: -Received: from hades.usol.com (IDENT:root@hades.usol.com [208.232.58.41]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5O1SuF06381 - for ; Sun, 23 Jun 2002 21:28:57 -0400 (EDT) -Received: from 01-072.024.popsite.net (01-072.024.popsite.net [216.126.160.72]) - by hades.usol.com (8.11.6/8.11.6) with ESMTP id g5O1Ssj09303; - Sun, 23 Jun 2002 21:28:55 -0400 -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -From: "J. R. Nield" -To: Bruce Momjian -cc: Curt Sampson , Tom Lane , - Michael Loftis - , mlw , - PostgreSQL Hacker - -In-Reply-To: <200206231936.g5NJasa07642@candle.pha.pa.us> -References: <200206231936.g5NJasa07642@candle.pha.pa.us> -Content-Type: text/plain -Content-Transfer-Encoding: 7bit -X-Mailer: Ximian Evolution 1.0.3 (1.0.3-6) -Date: 23 Jun 2002 21:29:23 -0400 -Message-ID: <1024882167.1793.733.camel@localhost.localdomain> -MIME-Version: 1.0 -Status: ROr - -On Sun, 2002-06-23 at 15:36, Bruce Momjian wrote: -> Yes, I don't see writing to two files vs. one to be any win, especially -> when we need to fsync both of them. What I would really like is to -> avoid the double I/O of writing to WAL and to the data file; improving -> that would be a huge win. -> - -If is impossible to do what you want. You can not protect against -partial writes without writing pages twice and calling fdatasync between -them while going through a generic filesystem. The best disk array will -not protect you if the operating system does not align block writes to -the structure of the underlying device. Even with raw devices, you need -special support or knowledge of the operating system and/or the disk -device to ensure that each write request will be atomic to the -underlying hardware. - -All other systems rely on the fact that you can recover a damaged file -using the log archive. This means downtime in the rare case, but no data -loss. Until PostgreSQL can do this, then it will not be acceptable for -real critical production use. This is not to knock PostgreSQL, because -it is a very good database system, and clearly the best open-source one. -It even has feature advantages over the commercial systems. But at the -end of the day, unless you have complete understanding of the I/O system -from write(2) through to the disk system, the only sure ways to protect -against partial writes are by "careful writes" (in the WAL log or -elsewhere, writing pages twice), or by requiring (and allowing) users to -do log-replay recovery when a file is corrupted by a partial write. As -long as there is a UPS, and the operating system doesn't crash, then -there still should be no partial writes. - -If we log pages to WAL, they are useless when archived (after a -checkpoint). So either we have a separate "log" for them (the ping-pong -file), or we should at least remove them when archived, which makes log -archiving more complex but is perfectly doable. - -Finally, I would love to hear why we are using the operating system -buffer manager at all. The OS is acting as a secondary buffer manager -for us. Why is that? What flaw in our I/O system does this reveal? I -know that: - ->We sync only WAL, not the other pages, except for the sync() call we do -> during checkpoint when we discard old WAL files. - -But this is probably not a good thing. We should only be writing blocks -when they need to be on disk. We should not be expecting the OS to write -them "sometime later" and avoid blocking (as long) for the write. If we -need that, then our buffer management is wrong and we need to fix it. -The reason we are doing this is because we expect the OS buffer manager -to do asynchronous I/O for us, but then we don't control the order. That -is the reason why we have to call fdatasync(), to create "sequence -points". - -The reason we have performance problems with either D_OSYNC or fdatasync -on the normal relations is because we have no dbflush process. This -causes an unacceptable amount of I/O blocking by other transactions. - -The ORACLE people were not kidding when they said that they could not -certify Linux for production use until it supported O_DSYNC. Can you -explain why that was the case? - -Finally, let me apologize if the above comes across as somewhat -belligerent. I know very well that I can't compete with you guys for -knowledge of the PosgreSQL system. I am still at a loss when I look at -the optimizer and executor modules, and it will take some time before I -can follow discussion of that area. Even then, I doubt my ability to -compare with people like Mr. Lane and Mr. Momjian in experience and -general intelligence, or in the field of database programming and -software development in particular. However, this discussion and a -search of the pgsql-hackers archives reveals this problem to be the KEY -area of PostgreSQL's failing, and general misunderstanding, when -compared to its commercial competitors. - -Sincerely, - - J. R. Nield - --- -J. R. Nield -jrnield@usol.com - - - - -From pgsql-hackers-owner+M24090@postgresql.org Mon Jun 24 12:38:04 2002 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5OGc3F05962 - for ; Mon, 24 Jun 2002 12:38:03 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP - id 81B9F4768DF; Mon, 24 Jun 2002 10:18:05 -0400 (EDT) -Mailbox-Line: From pgman@candle.pha.pa.us Mon Jun 24 10:18:05 2002 -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 81F08476473; Mon, 24 Jun 2002 08:55:28 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP id CDDFA475CC3 - for ; Mon, 24 Jun 2002 08:37:44 -0400 (EDT) -Mailbox-Line: From pgman@candle.pha.pa.us Mon Jun 24 08:37:44 2002 -Received: from candle.pha.pa.us (216-55-132-35.dsl.san-diego.abac.net [216.55.132.35]) - by postgresql.org (Postfix) with ESMTP id 5C971475858 - for ; Sun, 23 Jun 2002 22:47:12 -0400 (EDT) -Received: (from pgman@localhost) - by candle.pha.pa.us (8.11.6/8.10.1) id g5O2ki712992; - Sun, 23 Jun 2002 22:46:44 -0400 (EDT) -From: Bruce Momjian -Message-ID: <200206240246.g5O2ki712992@candle.pha.pa.us> -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: <1024882167.1793.733.camel@localhost.localdomain> -To: "J. R. Nield" -Date: Sun, 23 Jun 2002 22:46:44 -0400 (EDT) -cc: Curt Sampson , Tom Lane , - Michael Loftis , mlw , - PostgreSQL Hacker -X-Mailer: ELM [version 2.4ME+ PL97 (25)] -MIME-Version: 1.0 -Content-Transfer-Encoding: 7bit -Content-Type: text/plain; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Spam-Status: No, hits=-3.4 required=5.0 - tests=IN_REP_TO - version=2.30 -Status: RO - -J. R. Nield wrote: -> On Sun, 2002-06-23 at 15:36, Bruce Momjian wrote: -> > Yes, I don't see writing to two files vs. one to be any win, especially -> > when we need to fsync both of them. What I would really like is to -> > avoid the double I/O of writing to WAL and to the data file; improving -> > that would be a huge win. -> > -> -> If is impossible to do what you want. You can not protect against -> partial writes without writing pages twice and calling fdatasync between -> them while going through a generic filesystem. The best disk array will -> not protect you if the operating system does not align block writes to -> the structure of the underlying device. Even with raw devices, you need -> special support or knowledge of the operating system and/or the disk -> device to ensure that each write request will be atomic to the -> underlying hardware. - -Yes, I suspected it was impossible, but that doesn't mean I want it any -less. ;-) - -> All other systems rely on the fact that you can recover a damaged file -> using the log archive. This means downtime in the rare case, but no data -> loss. Until PostgreSQL can do this, then it will not be acceptable for -> real critical production use. This is not to knock PostgreSQL, because -> it is a very good database system, and clearly the best open-source one. -> It even has feature advantages over the commercial systems. But at the -> end of the day, unless you have complete understanding of the I/O system -> from write(2) through to the disk system, the only sure ways to protect -> against partial writes are by "careful writes" (in the WAL log or -> elsewhere, writing pages twice), or by requiring (and allowing) users to -> do log-replay recovery when a file is corrupted by a partial write. As -> long as there is a UPS, and the operating system doesn't crash, then -> there still should be no partial writes. - -You are talking point-in-time recovery, a major missing feature right -next to replication, and I agree it makes PostgreSQL unacceptable for -some applications. Point taken. - -And the interesting thing you are saying is that with point-in-time -recovery, we don't need to write pre-write images of pages because if we -detect a partial page write, we then abort the database and tell the -user to do a point-in-time recovery, basically meaning we are using the -previous full backup as our pre-write page image and roll forward using -the logical logs. This is clearly a nice thing to be able to do because -it let's you take a pre-write image of the page once during full backup, -keep it offline, and bring it back in the rare case of a full page write -failure. I now can see how the MSSQL tearoff-bits would be used, not -for recovery, but to detect a partial write and force a point-in-time -recovery from the administrator. - - -> If we log pages to WAL, they are useless when archived (after a -> checkpoint). So either we have a separate "log" for them (the ping-pong -> file), or we should at least remove them when archived, which makes log -> archiving more complex but is perfectly doable. - -Yes, that is how we will do point-in-time recovery; remove the -pre-write page images and archive the rest. It is more complex, but -having the fsync all in one file is too big a win. - -> Finally, I would love to hear why we are using the operating system -> buffer manager at all. The OS is acting as a secondary buffer manager -> for us. Why is that? What flaw in our I/O system does this reveal? I -> know that: -> -> >We sync only WAL, not the other pages, except for the sync() call we do -> > during checkpoint when we discard old WAL files. -> -> But this is probably not a good thing. We should only be writing blocks -> when they need to be on disk. We should not be expecting the OS to write -> them "sometime later" and avoid blocking (as long) for the write. If we -> need that, then our buffer management is wrong and we need to fix it. -> The reason we are doing this is because we expect the OS buffer manager -> to do asynchronous I/O for us, but then we don't control the order. That -> is the reason why we have to call fdatasync(), to create "sequence -> points". - -Yes. I think I understand. It is true we have to fsync WAL because we -can't control the individual writes by the OS. - -> The reason we have performance problems with either D_OSYNC or fdatasync -> on the normal relations is because we have no dbflush process. This -> causes an unacceptable amount of I/O blocking by other transactions. - -Uh, that would force writes all over the disk. Why do we really care how -the OS writes them? If we are going to fsync, let's just do the one -file and be done with it. What would a separate flusher process really -buy us if it has to use fsync too. The main backend doesn't have to -wait for the fsync, but then again, we can't say the transaction is -committed until it hits the disk, so how does a flusher help? - -> The ORACLE people were not kidding when they said that they could not -> certify Linux for production use until it supported O_DSYNC. Can you -> explain why that was the case? - -I don't see O_DSYNC as very different from write/fsync(or fdatasync). - -> Finally, let me apologize if the above comes across as somewhat -> belligerent. I know very well that I can't compete with you guys for -> knowledge of the PostgreSQL system. I am still at a loss when I look at -> the optimizer and executor modules, and it will take some time before I -> can follow discussion of that area. Even then, I doubt my ability to -> compare with people like Mr. Lane and Mr. Momjian in experience and -> general intelligence, or in the field of database programming and -> software development in particular. However, this discussion and a -> search of the pgsql-hackers archives reveals this problem to be the KEY -> area of PostgreSQL's failing, and general misunderstanding, when -> compared to its commercial competitors. - -We appreciate your ideas. Few of us are professional db folks so we are -always looking for good ideas. - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 853-3000 - + If your life is a hard drive, | 830 Blythe Avenue - + Christ can be your backup. | Drexel Hill, Pennsylvania 19026 - - - ----------------------------(end of broadcast)--------------------------- -TIP 6: Have you searched our list archives? - -http://archives.postgresql.org - - - -From cjs@cynic.net Sun Jun 23 23:40:59 2002 -Return-path: -Received: from academic.cynic.net (academic.cynic.net [63.144.177.3]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5O3evF17903 - for ; Sun, 23 Jun 2002 23:40:58 -0400 (EDT) -Received: from angelic-academic.cvpn.cynic.net (angelic-academic.cvpn.cynic.net [198.73.220.224]) - by academic.cynic.net (Postfix) with ESMTP - id 37F36F820; Mon, 24 Jun 2002 03:40:54 +0000 (UTC) -Date: Mon, 24 Jun 2002 12:40:51 +0900 (JST) -From: Curt Sampson -To: "J. R. Nield" -cc: Bruce Momjian , Tom Lane , - Michael Loftis , mlw , - PostgreSQL Hacker -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: <1024882167.1793.733.camel@localhost.localdomain> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Status: RO - -On 23 Jun 2002, J. R. Nield wrote: - -> If is impossible to do what you want. You can not protect against -> partial writes without writing pages twice and calling fdatasync -> between them while going through a generic filesystem. - -I agree with this. - -> The best disk array will not protect you if the operating system does -> not align block writes to the structure of the underlying device. - -This I don't quite understand. Assuming you're using a SCSI drive -(and this mostly applies to ATAPI/IDE, too), you can do naught but -align block writes to the structure of the underlying device. When you -initiate a SCSI WRITE command, you start by telling the device at which -block to start writing and how many blocks you intend to write. Then you -start passing the data. - -(See http://www.danbbs.dk/~dino/SCSI/SCSI2-09.html#9.2.21 for parameter -details for the SCSI WRITE(10) command. You may find the SCSI 2 -specification, at http://www.danbbs.dk/~dino/SCSI/ to be a useful -reference here.) - -> Even with raw devices, you need special support or knowledge of the -> operating system and/or the disk device to ensure that each write -> request will be atomic to the underlying hardware. - -Well, so here I guess you're talking about two things: - - 1. When you request, say, an 8K block write, will the OS really - write it to disk in a single 8K or multiple of 8K SCSI write - command? - - 2. Does the SCSI device you're writing to consider these writes to - be transactional. That is, if the write is interrupted before being - completed, does the SCSI device guarantee that the partially-sent - data is not written, and the old data is maintained? And of course, - does it guarantee that, when it acknowledges a write, that write is - now in stable storage and will never go away? - -Both of these are not hard to guarantee, actually. For a BSD-based OS, -for example, just make sure that your filesystem block size is the -same as or a multiple of the database block size. BSD will never write -anything other than a block or a sequence of blocks to a disk in a -single SCSI transaction (unless you've got a really odd SCSI driver). -And for your disk, buy a Baydel or Clarion disk array, or something -similar. - -Given that it's not hard to set up a system that meets these criteria, -and this is in fact commonly done for database servers, it would seem a -good idea for postgres to have the option to take advantage of the time -and money spent and adjust its performance upward appropriately. - -> All other systems rely on the fact that you can recover a damaged file -> using the log archive. - -Not exactly. For MS SQL Server, at any rate, if it detects a page tear -you cannot restore based on the log file alone. You need a full or -partial backup that includes that entire torn block. - -> This means downtime in the rare case, but no data loss. Until -> PostgreSQL can do this, then it will not be acceptable for real -> critical production use. - -It seems to me that it is doing this right now. In fact, it's more -reliable than some commerial systems (such as SQL Server) because it can -recover from a torn block with just the logfile. - -> But at the end of the day, unless you have complete understanding of -> the I/O system from write(2) through to the disk system, the only sure -> ways to protect against partial writes are by "careful writes" (in -> the WAL log or elsewhere, writing pages twice), or by requiring (and -> allowing) users to do log-replay recovery when a file is corrupted by -> a partial write. - -I don't understand how, without a copy of the old data that was in the -torn block, you can restore that block from just log file entries. Can -you explain this to me? Take, as an example, a block with ten tuples, -only one of which has been changed "recently." (I.e., only that change -is in the log files.) - -> If we log pages to WAL, they are useless when archived (after a -> checkpoint). So either we have a separate "log" for them (the -> ping-pong file), or we should at least remove them when archived, -> which makes log archiving more complex but is perfectly doable. - -Right. That seems to me a better option, since we've now got only one -write point on the disk rather than two. - -> Finally, I would love to hear why we are using the operating system -> buffer manager at all. The OS is acting as a secondary buffer manager -> for us. Why is that? What flaw in our I/O system does this reveal? - -It's acting as a "second-level" buffer manager, yes, but to say it's -"secondary" may be a bit misleading. On most of the systems I've set -up, the OS buffer cache is doing the vast majority of the work, and the -postgres buffering is fairly minimal. - -There are some good (and some perhaps not-so-good) reasons to do it this -way. I'll list them more or less in the order of best to worst: - - 1. The OS knows where the blocks physically reside on disk, and - postgres does not. Therefore it's in the interest of postgresql to - dispatch write responsibility back to the OS as quickly as possible - so that the OS can prioritize requests appropriately. Most operating - systems use an "elevator" algorithm to minimize disk head movement; - but if the OS does not have a block that it could write while the - head is "on the way" to another request, it can't write it in that - head pass. - - 2. Postgres does not know about any "bank-switching" tricks for - mapping more physical memory than it has address space. Thus, on - 32-bit machines, postgres might be limited to mapping 2 or 3 GB of - memory, even though the machine has, say, 6 GB of physical RAM. The - OS can use all of the available memory for caching; postgres cannot. - - 3. A lot of work has been put into the seek algorithms, read-ahead - algorithms, block allocation algorithms, etc. in the OS. Why - duplicate all that work again in postgres? - -When you say things like the following: - -> We should only be writing blocks when they need to be on disk. We -> should not be expecting the OS to write them "sometime later" and -> avoid blocking (as long) for the write. If we need that, then our -> buffer management is wrong and we need to fix it. - -you appear to be making the arugment that we should take the route of -other database systems, and use raw devices and our own management of -disk block allocation. If so, you might want first to look back through -the archives at the discussion I and several others had about this a -month or two ago. After looking in detail at what NetBSD, at least, does -in terms of its disk I/O algorithms and buffering, I've pretty much come -around, at least for the moment, to the attitude that we should stick -with using the OS. I wouldn't mind seeing postgres be able to manage all -of this stuff, but it's a *lot* of work for not all that much benefit -that I can see. - -> The ORACLE people were not kidding when they said that they could not -> certify Linux for production use until it supported O_DSYNC. Can you -> explain why that was the case? - -I'm suspecting it's because Linux at the time had no raw devices, so -O_DSYNC was the only other possible method of making sure that disk -writes actually got to disk. - -You certainly don't want to use O_DSYNC if you can use another method, -because O_DSYNC still goes through the the operating system's buffer -cache, wasting memory and double-caching things. If you're doing your -own management, you need either to use a raw device or open files with -the flag that indicates that the buffer cache should not be used at all -for reads from and writes to that file. - -> However, this discussion and a search of the pgsql-hackers archives -> reveals this problem to be the KEY area of PostgreSQL's failing, and -> general misunderstanding, when compared to its commercial competitors. - -No, I think it's just that you're under a few minor misapprehensions -here about what postgres and the OS are actually doing. As I said, I -went through this whole exact argument a month or two ago, on this very -list, and I came around to the idea that what postgres is doing now -works quite well, at least on NetBSD. (Most other OSes have disk I/O -algorithms that are pretty much as good or better.) There might be a -very slight advantage to doing all one's own I/O management, but it's -a huge amount of work, and I think that much effort could be much more -usefully applied to other areas. - -Just as a side note, I've been a NetBSD developer since about '96, -and have been delving into the details of OS design since well before -that time, so I'm coming to this with what I hope is reasonably good -knowledge of how disks work and how operating systems use them. (Not -that this should stop you from pointing out holes in my arguments. :-)) - -cjs --- -Curt Sampson +81 90 7737 2974 http://www.netbsd.org - Don't you know, in this new Dark Age, we're all light. --XTC - - -From pgsql-hackers-owner+M24112@postgresql.org Mon Jun 24 18:16:36 2002 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5OMGaF00910 - for ; Mon, 24 Jun 2002 18:16:36 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP - id A2EF1476475; Mon, 24 Jun 2002 16:43:38 -0400 (EDT) -Mailbox-Line: From tgl@sss.pgh.pa.us Mon Jun 24 16:43:38 2002 -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id BA57D476148; Mon, 24 Jun 2002 14:14:00 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP id 93D6A477214 - for ; Mon, 24 Jun 2002 13:59:17 -0400 (EDT) -Mailbox-Line: From tgl@sss.pgh.pa.us Mon Jun 24 13:59:17 2002 -Received: from sss.pgh.pa.us (unknown [192.204.191.242]) - by postgresql.org (Postfix) with ESMTP id D70AA476401 - for ; Mon, 24 Jun 2002 10:06:26 -0400 (EDT) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id g5OE6J117666; - Mon, 24 Jun 2002 10:06:19 -0400 (EDT) -To: Curt Sampson -cc: Bruce Momjian , "J. R. Nield" , - Michael Loftis , mlw , - PostgreSQL Hacker -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: -References: -Comments: In-reply-to Curt Sampson - message dated "Mon, 24 Jun 2002 09:09:30 +0900" -Date: Mon, 24 Jun 2002 10:06:19 -0400 -Message-ID: <17663.1024927579@sss.pgh.pa.us> -From: Tom Lane -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Spam-Status: No, hits=-5.3 required=5.0 - tests=IN_REP_TO,X_NOT_PRESENT - version=2.30 -Status: RO - -> On Sun, 23 Jun 2002, Bruce Momjian wrote: ->> Yes, I don't see writing to two files vs. one to be any win, especially ->> when we need to fsync both of them. What I would really like is to ->> avoid the double I/O of writing to WAL and to the data file; improving ->> that would be a huge win. - -I don't believe it's possible to eliminate the double I/O. Keep in mind -though that in the ideal case (plenty of shared buffers) you are only -paying two writes per modified block per checkpoint interval --- one to -the WAL during the first write of the interval, and then a write to the -real datafile issued by the checkpoint process. Anything that requires -transaction commits to write data blocks will likely result in more I/O -not less, at least for blocks that are modified by several successive -transactions. - -The only thing I've been able to think of that seems like it might -improve matters is to make the WAL writing logic aware of the layout -of buffer pages --- specifically, to know that our pages generally -contain an uninteresting "hole" in the middle, and not write the hole. -Optimistically this might reduce the WAL data volume by something -approaching 50%; though pessimistically (if most pages are near full) -it wouldn't help much. - -This was not very feasible when the WAL code was designed because the -buffer manager needed to cope with both normal pages and pg_log pages, -but as of 7.2 I think it'd be safe to assume that all pages have the -standard layout. - - regards, tom lane - - - ----------------------------(end of broadcast)--------------------------- -TIP 2: you can get off all lists at once with the unregister command - (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) - - - -From pgsql-hackers-owner+M24116@postgresql.org Mon Jun 24 20:32:07 2002 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5P0W7F10985 - for ; Mon, 24 Jun 2002 20:32:07 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP - id EBCE547632E; Mon, 24 Jun 2002 18:54:34 -0400 (EDT) -Mailbox-Line: From pgman@candle.pha.pa.us Mon Jun 24 18:54:34 2002 -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 3EB93476D85; Mon, 24 Jun 2002 17:12:18 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP id EBC20476E2E - for ; Mon, 24 Jun 2002 14:54:40 -0400 (EDT) -Mailbox-Line: From pgman@candle.pha.pa.us Mon Jun 24 14:54:40 2002 -Received: from candle.pha.pa.us (216-55-132-35.dsl.san-diego.abac.net [216.55.132.35]) - by postgresql.org (Postfix) with ESMTP id 1C8874760C2 - for ; Mon, 24 Jun 2002 12:40:53 -0400 (EDT) -Received: (from pgman@localhost) - by candle.pha.pa.us (8.11.6/8.10.1) id g5OGeVY06116; - Mon, 24 Jun 2002 12:40:31 -0400 (EDT) -From: Bruce Momjian -Message-ID: <200206241640.g5OGeVY06116@candle.pha.pa.us> -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: <17663.1024927579@sss.pgh.pa.us> -To: Tom Lane -Date: Mon, 24 Jun 2002 12:40:31 -0400 (EDT) -cc: Curt Sampson , "J. R. Nield" , - Michael Loftis , mlw , - PostgreSQL Hacker -X-Mailer: ELM [version 2.4ME+ PL97 (25)] -MIME-Version: 1.0 -Content-Transfer-Encoding: 7bit -Content-Type: text/plain; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Spam-Status: No, hits=-3.4 required=5.0 - tests=IN_REP_TO - version=2.30 -Status: RO - -Tom Lane wrote: -> > On Sun, 23 Jun 2002, Bruce Momjian wrote: -> >> Yes, I don't see writing to two files vs. one to be any win, especially -> >> when we need to fsync both of them. What I would really like is to -> >> avoid the double I/O of writing to WAL and to the data file; improving -> >> that would be a huge win. -> -> I don't believe it's possible to eliminate the double I/O. Keep in mind -> though that in the ideal case (plenty of shared buffers) you are only -> paying two writes per modified block per checkpoint interval --- one to -> the WAL during the first write of the interval, and then a write to the -> real datafile issued by the checkpoint process. Anything that requires -> transaction commits to write data blocks will likely result in more I/O -> not less, at least for blocks that are modified by several successive -> transactions. -> -> The only thing I've been able to think of that seems like it might -> improve matters is to make the WAL writing logic aware of the layout -> of buffer pages --- specifically, to know that our pages generally -> contain an uninteresting "hole" in the middle, and not write the hole. -> Optimistically this might reduce the WAL data volume by something -> approaching 50%; though pessimistically (if most pages are near full) -> it wouldn't help much. - -Good idea. How about putting the page through or TOAST compression -routine before writing it to WAL? Should be pretty easy and fast and -doesn't require any knowledge of the page format. - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 853-3000 - + If your life is a hard drive, | 830 Blythe Avenue - + Christ can be your backup. | Drexel Hill, Pennsylvania 19026 - - - ----------------------------(end of broadcast)--------------------------- -TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org - - - -From pgsql-hackers-owner+M24114@postgresql.org Mon Jun 24 17:54:35 2002 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5OLsZF28642 - for ; Mon, 24 Jun 2002 17:54:35 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP - id BD68F47683C; Mon, 24 Jun 2002 16:46:24 -0400 (EDT) -Mailbox-Line: From tgl@sss.pgh.pa.us Mon Jun 24 16:46:24 2002 -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id B2719476B31; Mon, 24 Jun 2002 16:01:51 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP id 950004770BC - for ; Mon, 24 Jun 2002 14:59:46 -0400 (EDT) -Mailbox-Line: From tgl@sss.pgh.pa.us Mon Jun 24 14:59:46 2002 -Received: from sss.pgh.pa.us (unknown [192.204.191.242]) - by postgresql.org (Postfix) with ESMTP id A0756475BB7 - for ; Mon, 24 Jun 2002 13:11:41 -0400 (EDT) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id g5OHB1119826; - Mon, 24 Jun 2002 13:11:02 -0400 (EDT) -To: Bruce Momjian -cc: Curt Sampson , "J. R. Nield" , - Michael Loftis , mlw , - PostgreSQL Hacker -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: <200206241640.g5OGeVY06116@candle.pha.pa.us> -References: <200206241640.g5OGeVY06116@candle.pha.pa.us> -Comments: In-reply-to Bruce Momjian - message dated "Mon, 24 Jun 2002 12:40:31 -0400" -Date: Mon, 24 Jun 2002 13:11:01 -0400 -Message-ID: <19823.1024938661@sss.pgh.pa.us> -From: Tom Lane -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Spam-Status: No, hits=-5.3 required=5.0 - tests=IN_REP_TO,X_NOT_PRESENT - version=2.30 -Status: RO - -Bruce Momjian writes: ->> The only thing I've been able to think of that seems like it might ->> improve matters is to make the WAL writing logic aware of the layout ->> of buffer pages --- specifically, to know that our pages generally ->> contain an uninteresting "hole" in the middle, and not write the hole. ->> Optimistically this might reduce the WAL data volume by something ->> approaching 50%; though pessimistically (if most pages are near full) ->> it wouldn't help much. - -> Good idea. How about putting the page through or TOAST compression -> routine before writing it to WAL? Should be pretty easy and fast and -> doesn't require any knowledge of the page format. - -Easy, maybe, but fast definitely NOT. The compressor is not speedy. -Given that we have to be holding various locks while we build WAL -records, I do not think it's a good idea to add CPU time there. - -Also, compressing already-compressed data is not a win ... - - regards, tom lane - - - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate -subscribe-nomail command to majordomo@postgresql.org so that your -message can get through to the mailing list cleanly - - - -From jrnield@usol.com Mon Jun 24 16:49:25 2002 -Return-path: -Received: from hades.usol.com (IDENT:root@hades.usol.com [208.232.58.41]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5OKnNF23393 - for ; Mon, 24 Jun 2002 16:49:24 -0400 (EDT) -Received: from 08-113.024.popsite.net (08-113.024.popsite.net [66.19.4.113]) - by hades.usol.com (8.11.6/8.11.6) with ESMTP id g5OKnHV19100; - Mon, 24 Jun 2002 16:49:18 -0400 -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -From: "J. R. Nield" -To: Curt Sampson -cc: Bruce Momjian , Tom Lane , - PostgreSQL Hacker -In-Reply-To: -References: -Content-Type: text/plain -Content-Transfer-Encoding: 7bit -X-Mailer: Ximian Evolution 1.0.3 (1.0.3-6) -Date: 24 Jun 2002 16:49:42 -0400 -Message-ID: <1024951786.1793.865.camel@localhost.localdomain> -MIME-Version: 1.0 -Status: ROr - -On Sun, 2002-06-23 at 23:40, Curt Sampson wrote: -> On 23 Jun 2002, J. R. Nield wrote: -> -> > If is impossible to do what you want. You can not protect against -> > partial writes without writing pages twice and calling fdatasync -> > between them while going through a generic filesystem. -> -> I agree with this. -> -> > The best disk array will not protect you if the operating system does -> > not align block writes to the structure of the underlying device. -> -> This I don't quite understand. Assuming you're using a SCSI drive -> (and this mostly applies to ATAPI/IDE, too), you can do naught but -> align block writes to the structure of the underlying device. When you -> initiate a SCSI WRITE command, you start by telling the device at which -> block to start writing and how many blocks you intend to write. Then you -> start passing the data. -> - -All I'm saying is that the entire postgresql block write must be -converted into exactly one SCSI write command in all cases, and I don't -know a portable way to ensure this. - -> > Even with raw devices, you need special support or knowledge of the -> > operating system and/or the disk device to ensure that each write -> > request will be atomic to the underlying hardware. -> -> Well, so here I guess you're talking about two things: -> -> 1. When you request, say, an 8K block write, will the OS really -> write it to disk in a single 8K or multiple of 8K SCSI write -> command? -> -> 2. Does the SCSI device you're writing to consider these writes to -> be transactional. That is, if the write is interrupted before being -> completed, does the SCSI device guarantee that the partially-sent -> data is not written, and the old data is maintained? And of course, -> does it guarantee that, when it acknowledges a write, that write is -> now in stable storage and will never go away? -> -> Both of these are not hard to guarantee, actually. For a BSD-based OS, -> for example, just make sure that your filesystem block size is the -> same as or a multiple of the database block size. BSD will never write -> anything other than a block or a sequence of blocks to a disk in a -> single SCSI transaction (unless you've got a really odd SCSI driver). -> And for your disk, buy a Baydel or Clarion disk array, or something -> similar. -> -> Given that it's not hard to set up a system that meets these criteria, -> and this is in fact commonly done for database servers, it would seem a -> good idea for postgres to have the option to take advantage of the time -> and money spent and adjust its performance upward appropriately. - -I agree with this. My point was only that you need to know what -guarantees your operating system/hardware combination provides on a -case-by-case basis, and there is no standard way for a program to -discover this. Most system administrators are not going to know this -either, unless databases are their main responsibility. - -> -> > All other systems rely on the fact that you can recover a damaged file -> > using the log archive. -> -> Not exactly. For MS SQL Server, at any rate, if it detects a page tear -> you cannot restore based on the log file alone. You need a full or -> partial backup that includes that entire torn block. -> - -I should have been more specific: you need a backup of the file from -some time ago, plus all the archived logs from then until the current -log sequence number. - -> > This means downtime in the rare case, but no data loss. Until -> > PostgreSQL can do this, then it will not be acceptable for real -> > critical production use. -> -> It seems to me that it is doing this right now. In fact, it's more -> reliable than some commerial systems (such as SQL Server) because it can -> recover from a torn block with just the logfile. - -Again, what I meant to say is that the commercial systems can recover -with an old file backup + logs. How old the backup can be depends only -on how much time you are willing to spend playing the logs forward. So -if you do a full backup once a week, and multiplex and backup the logs, -then even if a backup tape gets destroyed you can still survive. It just -takes longer. - -Also, postgreSQL can't recover from any other type of block corruption, -while the commercial systems can. That's what I meant by the "critical -production use" comment, which was sort-of unfair. - -So I would say they are equally reliable for torn pages (but not bad -blocks), and the commercial systems let you trade potential recovery -time for not having to write the blocks twice. You do need to back-up -the log archives though. - -> -> > But at the end of the day, unless you have complete understanding of -> > the I/O system from write(2) through to the disk system, the only sure -> > ways to protect against partial writes are by "careful writes" (in -> > the WAL log or elsewhere, writing pages twice), or by requiring (and -> > allowing) users to do log-replay recovery when a file is corrupted by -> > a partial write. -> -> I don't understand how, without a copy of the old data that was in the -> torn block, you can restore that block from just log file entries. Can -> you explain this to me? Take, as an example, a block with ten tuples, -> only one of which has been changed "recently." (I.e., only that change -> is in the log files.) -> -> -> > If we log pages to WAL, they are useless when archived (after a -> > checkpoint). So either we have a separate "log" for them (the -> > ping-pong file), or we should at least remove them when archived, -> > which makes log archiving more complex but is perfectly doable. -> -> Right. That seems to me a better option, since we've now got only one -> write point on the disk rather than two. - -OK. I agree with this now. - -> -> > Finally, I would love to hear why we are using the operating system -> > buffer manager at all. The OS is acting as a secondary buffer manager -> > for us. Why is that? What flaw in our I/O system does this reveal? -> -> It's acting as a "second-level" buffer manager, yes, but to say it's -> "secondary" may be a bit misleading. On most of the systems I've set -> up, the OS buffer cache is doing the vast majority of the work, and the -> postgres buffering is fairly minimal. -> -> There are some good (and some perhaps not-so-good) reasons to do it this -> way. I'll list them more or less in the order of best to worst: -> -> 1. The OS knows where the blocks physically reside on disk, and -> postgres does not. Therefore it's in the interest of postgresql to -> dispatch write responsibility back to the OS as quickly as possible -> so that the OS can prioritize requests appropriately. Most operating -> systems use an "elevator" algorithm to minimize disk head movement; -> but if the OS does not have a block that it could write while the -> head is "on the way" to another request, it can't write it in that -> head pass. -> -> 2. Postgres does not know about any "bank-switching" tricks for -> mapping more physical memory than it has address space. Thus, on -> 32-bit machines, postgres might be limited to mapping 2 or 3 GB of -> memory, even though the machine has, say, 6 GB of physical RAM. The -> OS can use all of the available memory for caching; postgres cannot. -> -> 3. A lot of work has been put into the seek algorithms, read-ahead -> algorithms, block allocation algorithms, etc. in the OS. Why -> duplicate all that work again in postgres? -> -> When you say things like the following: -> -> > We should only be writing blocks when they need to be on disk. We -> > should not be expecting the OS to write them "sometime later" and -> > avoid blocking (as long) for the write. If we need that, then our -> > buffer management is wrong and we need to fix it. -> -> you appear to be making the arugment that we should take the route of -> other database systems, and use raw devices and our own management of -> disk block allocation. If so, you might want first to look back through -> the archives at the discussion I and several others had about this a -> month or two ago. After looking in detail at what NetBSD, at least, does -> in terms of its disk I/O algorithms and buffering, I've pretty much come -> around, at least for the moment, to the attitude that we should stick -> with using the OS. I wouldn't mind seeing postgres be able to manage all -> of this stuff, but it's a *lot* of work for not all that much benefit -> that I can see. - -I'll back off on that. I don't know if we want to use the OS buffer -manager, but shouldn't we try to have our buffer manager group writes -together by files, and pro-actively get them out to disk? Right now, it -looks like all our write requests are delayed as long as possible and -the order in which they are written is pretty-much random, as is the -backend that writes the block, so there is no locality of reference even -when the blocks are adjacent on disk, and the write calls are spread-out -over all the backends. - -Would it not be the case that things like read-ahead, grouping writes, -and caching written data are probably best done by PostgreSQL, because -only our buffer manager can understand when they will be useful or when -they will thrash the cache? - -I may likely be wrong on this, and I haven't done any performance -testing. I shouldn't have brought this up alongside the logging issues, -but there seemed to be some question about whether the OS was actually -doing all these things behind the scene. - - -> -> > The ORACLE people were not kidding when they said that they could not -> > certify Linux for production use until it supported O_DSYNC. Can you -> > explain why that was the case? -> -> I'm suspecting it's because Linux at the time had no raw devices, so -> O_DSYNC was the only other possible method of making sure that disk -> writes actually got to disk. -> -> You certainly don't want to use O_DSYNC if you can use another method, -> because O_DSYNC still goes through the the operating system's buffer -> cache, wasting memory and double-caching things. If you're doing your -> own management, you need either to use a raw device or open files with -> the flag that indicates that the buffer cache should not be used at all -> for reads from and writes to that file. - -Would O_DSYNC|O_RSYNC turn off the cache? - -> -> > However, this discussion and a search of the pgsql-hackers archives -> > reveals this problem to be the KEY area of PostgreSQL's failing, and -> > general misunderstanding, when compared to its commercial competitors. -> -> No, I think it's just that you're under a few minor misapprehensions -> here about what postgres and the OS are actually doing. As I said, I -> went through this whole exact argument a month or two ago, on this very -> list, and I came around to the idea that what postgres is doing now -> works quite well, at least on NetBSD. (Most other OSes have disk I/O -> algorithms that are pretty much as good or better.) There might be a -> very slight advantage to doing all one's own I/O management, but it's -> a huge amount of work, and I think that much effort could be much more -> usefully applied to other areas. - -I will look for that discussion in the archives. - -The logging issue is a key one I think. At least I would be very nervous -as a DBA if I were running a system where any damaged file would cause -data loss. - -Does anyone know what the major barriers to infinite log replay are in -PostgreSQL? I'm trying to look for everything that might need to be -changed outside xlog.c, but surely this has come up before. Searching -the archives hasn't revealed much. - - - -As to the I/O issue: - -Since you know a lot about NetBSD internals, I'd be interested in -hearing about what postgresql looks like to the NetBSD buffer manager. -Am I right that strings of successive writes get randomized? What do our -cache-hit percentages look like? I'm going to do some experimenting with -this. - -> -> Just as a side note, I've been a NetBSD developer since about '96, -> and have been delving into the details of OS design since well before -> that time, so I'm coming to this with what I hope is reasonably good -> knowledge of how disks work and how operating systems use them. (Not -> that this should stop you from pointing out holes in my arguments. :-)) -> - -This stuff is very difficult to get right. Glad to know you follow this -list. - - -> cjs -> -- -> Curt Sampson +81 90 7737 2974 http://www.netbsd.org -> Don't you know, in this new Dark Age, we're all light. --XTC -> --- -J. R. Nield -jrnield@usol.com - - - - -From tgl@sss.pgh.pa.us Mon Jun 24 17:16:06 2002 -Return-path: -Received: from sss.pgh.pa.us (root@[192.204.191.242]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5OLG5F25284 - for ; Mon, 24 Jun 2002 17:16:05 -0400 (EDT) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id g5OLG2121379; - Mon, 24 Jun 2002 17:16:02 -0400 (EDT) -To: "J. R. Nield" -cc: Curt Sampson , Bruce Momjian , - PostgreSQL Hacker -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: <1024951786.1793.865.camel@localhost.localdomain> -References: <1024951786.1793.865.camel@localhost.localdomain> -Comments: In-reply-to "J. R. Nield" - message dated "24 Jun 2002 16:49:42 -0400" -Date: Mon, 24 Jun 2002 17:16:01 -0400 -Message-ID: <21376.1024953361@sss.pgh.pa.us> -From: Tom Lane -Status: RO - -"J. R. Nield" writes: -> Also, postgreSQL can't recover from any other type of block corruption, -> while the commercial systems can. - -Say again? - -> Would it not be the case that things like read-ahead, grouping writes, -> and caching written data are probably best done by PostgreSQL, because -> only our buffer manager can understand when they will be useful or when -> they will thrash the cache? - -I think you have been missing the point. No one denies that there will -be some incremental gain if we do all that. However, the conclusion of -everyone who has thought much about it (and I see Curt has joined that -group) is that the effort would be far out of proportion to the probable -gain. There are a lot of other things we desperately need to spend time -on that would not amount to re-engineering large quantities of OS-level -code. Given that most Unixen have perfectly respectable disk management -subsystems, we prefer to tune our code to make use of that stuff, rather -than follow the "conventional wisdom" that databases need to bypass it. - -Oracle can afford to do that sort of thing because they have umpteen -thousand developers available. Postgres does not. - - regards, tom lane - -From pgsql-hackers-owner+M24128@postgresql.org Mon Jun 24 22:01:58 2002 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5P21vF19918 - for ; Mon, 24 Jun 2002 22:01:57 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP - id 540B8475B33; Mon, 24 Jun 2002 21:34:40 -0400 (EDT) -Mailbox-Line: From pgman@candle.pha.pa.us Mon Jun 24 21:34:40 2002 -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 0A13F476965; Mon, 24 Jun 2002 19:30:14 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP id B4F62476E4A - for ; Mon, 24 Jun 2002 18:53:59 -0400 (EDT) -Mailbox-Line: From pgman@candle.pha.pa.us Mon Jun 24 18:53:59 2002 -Received: from candle.pha.pa.us (216-55-132-35.dsl.san-diego.abac.net [216.55.132.35]) - by postgresql.org (Postfix) with ESMTP id 36043475BF6 - for ; Mon, 24 Jun 2002 17:25:28 -0400 (EDT) -Received: (from pgman@localhost) - by candle.pha.pa.us (8.11.6/8.10.1) id g5OLPFG26140; - Mon, 24 Jun 2002 17:25:15 -0400 (EDT) -From: Bruce Momjian -Message-ID: <200206242125.g5OLPFG26140@candle.pha.pa.us> -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: <1024951786.1793.865.camel@localhost.localdomain> -To: "J. R. Nield" -Date: Mon, 24 Jun 2002 17:25:14 -0400 (EDT) -cc: Curt Sampson , Tom Lane , - PostgreSQL Hacker -X-Mailer: ELM [version 2.4ME+ PL97 (25)] -MIME-Version: 1.0 -Content-Transfer-Encoding: 7bit -Content-Type: text/plain; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Spam-Status: No, hits=-3.4 required=5.0 - tests=IN_REP_TO - version=2.30 -Status: RO - -J. R. Nield wrote: -> > This I don't quite understand. Assuming you're using a SCSI drive -> > (and this mostly applies to ATAPI/IDE, too), you can do naught but -> > align block writes to the structure of the underlying device. When you -> > initiate a SCSI WRITE command, you start by telling the device at which -> > block to start writing and how many blocks you intend to write. Then you -> > start passing the data. -> > -> -> All I'm saying is that the entire postgresql block write must be -> converted into exactly one SCSI write command in all cases, and I don't -> know a portable way to ensure this. - -... - -> I agree with this. My point was only that you need to know what -> guarantees your operating system/hardware combination provides on a -> case-by-case basis, and there is no standard way for a program to -> discover this. Most system administrators are not going to know this -> either, unless databases are their main responsibility. - -Yes, agreed. >1% are going to know the answer to this question so we -have to assume worst case. - -> > It seems to me that it is doing this right now. In fact, it's more -> > reliable than some commerial systems (such as SQL Server) because it can -> > recover from a torn block with just the logfile. -> -> Again, what I meant to say is that the commercial systems can recover -> with an old file backup + logs. How old the backup can be depends only -> on how much time you are willing to spend playing the logs forward. So -> if you do a full backup once a week, and multiplex and backup the logs, -> then even if a backup tape gets destroyed you can still survive. It just -> takes longer. -> -> Also, postgreSQL can't recover from any other type of block corruption, -> while the commercial systems can. That's what I meant by the "critical -> production use" comment, which was sort-of unfair. -> -> So I would say they are equally reliable for torn pages (but not bad -> blocks), and the commercial systems let you trade potential recovery -> time for not having to write the blocks twice. You do need to back-up -> the log archives though. - -Yes, good tradeoff analysis. We recover from partial writes quicker, -and don't require saving of log files, _but_ we don't recover from bad -disk blocks. Good summary. - -> I'll back off on that. I don't know if we want to use the OS buffer -> manager, but shouldn't we try to have our buffer manager group writes -> together by files, and pro-actively get them out to disk? Right now, it -> looks like all our write requests are delayed as long as possible and -> the order in which they are written is pretty-much random, as is the -> backend that writes the block, so there is no locality of reference even -> when the blocks are adjacent on disk, and the write calls are spread-out -> over all the backends. -> -> Would it not be the case that things like read-ahead, grouping writes, -> and caching written data are probably best done by PostgreSQL, because -> only our buffer manager can understand when they will be useful or when -> they will thrash the cache? - -The OS should handle all of this. We are doing main table writes but no -sync until checkpoint, so the OS can keep those blocks around and write -them at its convenience. It knows the size of the buffer cache and when -stuff is forced to disk. We can't second-guess that. - -> I may likely be wrong on this, and I haven't done any performance -> testing. I shouldn't have brought this up alongside the logging issues, -> but there seemed to be some question about whether the OS was actually -> doing all these things behind the scene. - -It had better. Looking at the kernel source is the way to know. - -> Does anyone know what the major barriers to infinite log replay are in -> PostgreSQL? I'm trying to look for everything that might need to be -> changed outside xlog.c, but surely this has come up before. Searching -> the archives hasn't revealed much. - -This has been brought up. Could we just save WAL files and get replay? -I believe some things have to be added to WAL to allow this, but it -seems possible. However, the pg_dump is just a data dump and does not -have the file offsets and things. Somehow you would need a tar-type -backup of the database, and with a running db, it is hard to get a valid -snapshot of that. - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 853-3000 - + If your life is a hard drive, | 830 Blythe Avenue - + Christ can be your backup. | Drexel Hill, Pennsylvania 19026 - - - ----------------------------(end of broadcast)--------------------------- -TIP 3: if posting/reading through Usenet, please send an appropriate -subscribe-nomail command to majordomo@postgresql.org so that your -message can get through to the mailing list cleanly - - - -From tgl@sss.pgh.pa.us Mon Jun 24 17:31:57 2002 -Return-path: -Received: from sss.pgh.pa.us (root@[192.204.191.242]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5OLVuF26684 - for ; Mon, 24 Jun 2002 17:31:56 -0400 (EDT) -Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) - by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id g5OLVu121485; - Mon, 24 Jun 2002 17:31:56 -0400 (EDT) -To: Bruce Momjian -cc: "J. R. Nield" , Curt Sampson , - PostgreSQL Hacker -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: <200206242125.g5OLPFG26140@candle.pha.pa.us> -References: <200206242125.g5OLPFG26140@candle.pha.pa.us> -Comments: In-reply-to Bruce Momjian - message dated "Mon, 24 Jun 2002 17:25:14 -0400" -Date: Mon, 24 Jun 2002 17:31:56 -0400 -Message-ID: <21482.1024954316@sss.pgh.pa.us> -From: Tom Lane -Status: ROr - -Bruce Momjian writes: ->> Does anyone know what the major barriers to infinite log replay are in ->> PostgreSQL? I'm trying to look for everything that might need to be ->> changed outside xlog.c, but surely this has come up before. Searching ->> the archives hasn't revealed much. - -> This has been brought up. Could we just save WAL files and get replay? -> I believe some things have to be added to WAL to allow this, but it -> seems possible. - -The Red Hat group has been looking at this somewhat; so far there seem -to be some minor tweaks that would be needed, but no showstoppers. - -> Somehow you would need a tar-type -> backup of the database, and with a running db, it is hard to get a valid -> snapshot of that. - -But you don't *need* a "valid snapshot", only a correct copy of -every block older than the first checkpoint in your WAL log series. -Any inconsistencies in your tar dump will look like repairable damage; -replaying the WAL log will fix 'em. - - regards, tom lane - -From pgsql-hackers-owner+M24133@postgresql.org Mon Jun 24 22:19:55 2002 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5P2JsF21543 - for ; Mon, 24 Jun 2002 22:19:54 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP - id 42391476E53; Mon, 24 Jun 2002 22:09:49 -0400 (EDT) -Mailbox-Line: From pgman@candle.pha.pa.us Mon Jun 24 22:09:49 2002 -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 191654774EB; Mon, 24 Jun 2002 20:26:08 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP id 8EB90476101 - for ; Mon, 24 Jun 2002 19:43:19 -0400 (EDT) -Mailbox-Line: From pgman@candle.pha.pa.us Mon Jun 24 19:43:19 2002 -Received: from candle.pha.pa.us (216-55-132-35.dsl.san-diego.abac.net [216.55.132.35]) - by postgresql.org (Postfix) with ESMTP id 08018476931 - for ; Mon, 24 Jun 2002 17:33:53 -0400 (EDT) -Received: (from pgman@localhost) - by candle.pha.pa.us (8.11.6/8.10.1) id g5OLXhl26908; - Mon, 24 Jun 2002 17:33:43 -0400 (EDT) -From: Bruce Momjian -Message-ID: <200206242133.g5OLXhl26908@candle.pha.pa.us> -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: <21482.1024954316@sss.pgh.pa.us> -To: Tom Lane -Date: Mon, 24 Jun 2002 17:33:43 -0400 (EDT) -cc: "J. R. Nield" , Curt Sampson , - PostgreSQL Hacker -X-Mailer: ELM [version 2.4ME+ PL97 (25)] -MIME-Version: 1.0 -Content-Transfer-Encoding: 7bit -Content-Type: text/plain; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Spam-Status: No, hits=-3.4 required=5.0 - tests=IN_REP_TO - version=2.30 -Status: RO - -Tom Lane wrote: -> Bruce Momjian writes: -> >> Does anyone know what the major barriers to infinite log replay are in -> >> PostgreSQL? I'm trying to look for everything that might need to be -> >> changed outside xlog.c, but surely this has come up before. Searching -> >> the archives hasn't revealed much. -> -> > This has been brought up. Could we just save WAL files and get replay? -> > I believe some things have to be added to WAL to allow this, but it -> > seems possible. -> -> The Red Hat group has been looking at this somewhat; so far there seem -> to be some minor tweaks that would be needed, but no showstoppers. - - -Good. - -> > Somehow you would need a tar-type -> > backup of the database, and with a running db, it is hard to get a valid -> > snapshot of that. -> -> But you don't *need* a "valid snapshot", only a correct copy of -> every block older than the first checkpoint in your WAL log series. -> Any inconsistencies in your tar dump will look like repairable damage; -> replaying the WAL log will fix 'em. - -Yes, my point was that you need physical file backups, not pg_dump, and -you have to be tricky about the files changing during the backup. You -_can_ work around changes to the files during backup. - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 853-3000 - + If your life is a hard drive, | 830 Blythe Avenue - + Christ can be your backup. | Drexel Hill, Pennsylvania 19026 -/usr/local/bin/mime: cannot create /dev/ttyp3: permission denied - - - ----------------------------(end of broadcast)--------------------------- -TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org - - - -From jrnield@usol.com Mon Jun 24 20:27:45 2002 -Return-path: -Received: from hades.usol.com (IDENT:root@hades.usol.com [208.232.58.41]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5P0RhF10711 - for ; Mon, 24 Jun 2002 20:27:44 -0400 (EDT) -Received: from 08-159.024.popsite.net (08-159.024.popsite.net [66.19.4.159]) - by hades.usol.com (8.11.6/8.11.6) with ESMTP id g5P0RbV01261; - Mon, 24 Jun 2002 20:27:37 -0400 -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -From: "J. R. Nield" -To: Tom Lane -cc: Curt Sampson , Bruce Momjian , - PostgreSQL Hacker -In-Reply-To: <21376.1024953361@sss.pgh.pa.us> -References: - <1024951786.1793.865.camel@localhost.localdomain> - <21376.1024953361@sss.pgh.pa.us> -Content-Type: text/plain -Content-Transfer-Encoding: 7bit -X-Mailer: Ximian Evolution 1.0.3 (1.0.3-6) -Date: 24 Jun 2002 20:28:00 -0400 -Message-ID: <1024964884.3031.876.camel@localhost.localdomain> -MIME-Version: 1.0 -Status: RO - -On Mon, 2002-06-24 at 17:16, Tom Lane wrote: - -> I think you have been missing the point... -Yes, this appears to be the case. Thanks especially to Curt for clearing -things up for me. - --- -J. R. Nield -jrnield@usol.com - - - - -From cjs@cynic.net Mon Jun 24 23:32:23 2002 -Return-path: -Received: from academic.cynic.net ([63.144.177.3]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5P3WMF28287 - for ; Mon, 24 Jun 2002 23:32:23 -0400 (EDT) -Received: from angelic-academic.cvpn.cynic.net (angelic-academic.cvpn.cynic.net [198.73.220.224]) - by academic.cynic.net (Postfix) with ESMTP - id 28AB5F820; Tue, 25 Jun 2002 03:32:08 +0000 (UTC) -Date: Tue, 25 Jun 2002 12:32:05 +0900 (JST) -From: Curt Sampson -To: "J. R. Nield" -cc: Bruce Momjian , Tom Lane , - PostgreSQL Hacker -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: <1024951786.1793.865.camel@localhost.localdomain> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Status: RO - -On 24 Jun 2002, J. R. Nield wrote: - -> All I'm saying is that the entire postgresql block write must be -> converted into exactly one SCSI write command in all cases, and I don't -> know a portable way to ensure this. - -No, there's no portable way. All you can do is give the admin who -is able to set things up safely the ability to turn of the now-unneeded -(and expensive) safety-related stuff that postgres does. - -> I agree with this. My point was only that you need to know what -> guarantees your operating system/hardware combination provides on a -> case-by-case basis, and there is no standard way for a program to -> discover this. Most system administrators are not going to know this -> either, unless databases are their main responsibility. - -Certainly this is true of pretty much every database system out there. - -cjs --- -Curt Sampson +81 90 7737 2974 http://www.netbsd.org - Don't you know, in this new Dark Age, we're all light. --XTC - - -From cjs@cynic.net Tue Jun 25 01:09:02 2002 -Return-path: -Received: from academic.cynic.net (academic.cynic.net [63.144.177.3]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5P591F07292 - for ; Tue, 25 Jun 2002 01:09:01 -0400 (EDT) -Received: from angelic-academic.cvpn.cynic.net (angelic-academic.cvpn.cynic.net [198.73.220.224]) - by academic.cynic.net (Postfix) with ESMTP - id 517BEF820; Tue, 25 Jun 2002 05:09:02 +0000 (UTC) -Date: Tue, 25 Jun 2002 14:08:59 +0900 (JST) -From: Curt Sampson -To: Tom Lane -cc: "J. R. Nield" , Bruce Momjian , - PostgreSQL Hacker -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: <21376.1024953361@sss.pgh.pa.us> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Status: ROr - -On Mon, 24 Jun 2002, Tom Lane wrote: - -> There are a lot of other things we desperately need to spend time -> on that would not amount to re-engineering large quantities of OS-level -> code. Given that most Unixen have perfectly respectable disk management -> subsystems, we prefer to tune our code to make use of that stuff, rather -> than follow the "conventional wisdom" that databases need to bypass it. -> ... -> Oracle can afford to do that sort of thing because they have umpteen -> thousand developers available. Postgres does not. - -Well, Oracle also started out, a long long time ago, on systems without -unified buffer cache and so on, and so they *had* to write this stuff -because otherwise data would not be cached. So Oracle can also afford to -maintain it now because the code already exists. - -cjs --- -Curt Sampson +81 90 7737 2974 http://www.netbsd.org - Don't you know, in this new Dark Age, we're all light. --XTC - - -From pgsql-hackers-owner+M24154@postgresql.org Tue Jun 25 09:22:38 2002 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id g5PDMbF03932 - for ; Tue, 25 Jun 2002 09:22:37 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP - id C12C3475E4A; Tue, 25 Jun 2002 09:22:32 -0400 (EDT) -Mailbox-Line: From pgman@candle.pha.pa.us Tue Jun 25 09:22:32 2002 -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 65471475C7A; Tue, 25 Jun 2002 09:22:23 -0400 (EDT) -Received: from localhost.localdomain (postgresql.org [64.49.215.8]) - by localhost (Postfix) with ESMTP id 97C8C475A7C - for ; Tue, 25 Jun 2002 09:22:20 -0400 (EDT) -Mailbox-Line: From pgman@candle.pha.pa.us Tue Jun 25 09:22:20 2002 -Received: from candle.pha.pa.us (216-55-132-35.dsl.san-diego.abac.net [216.55.132.35]) - by postgresql.org (Postfix) with ESMTP id 42C0B475A64 - for ; Tue, 25 Jun 2002 09:22:19 -0400 (EDT) -Received: (from pgman@localhost) - by candle.pha.pa.us (8.11.6/8.10.1) id g5PDM5B03772; - Tue, 25 Jun 2002 09:22:05 -0400 (EDT) -From: Bruce Momjian -Message-ID: <200206251322.g5PDM5B03772@candle.pha.pa.us> -Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE -In-Reply-To: -To: Curt Sampson -Date: Tue, 25 Jun 2002 09:22:05 -0400 (EDT) -cc: Tom Lane , "J. R. Nield" , - PostgreSQL Hacker -X-Mailer: ELM [version 2.4ME+ PL97 (25)] -MIME-Version: 1.0 -Content-Transfer-Encoding: 7bit -Content-Type: text/plain; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Spam-Status: No, hits=-3.4 required=5.0 - tests=IN_REP_TO - version=2.30 -Status: RO - -Curt Sampson wrote: -> On Mon, 24 Jun 2002, Tom Lane wrote: -> -> > There are a lot of other things we desperately need to spend time -> > on that would not amount to re-engineering large quantities of OS-level -> > code. Given that most Unixen have perfectly respectable disk management -> > subsystems, we prefer to tune our code to make use of that stuff, rather -> > than follow the "conventional wisdom" that databases need to bypass it. -> > ... -> > Oracle can afford to do that sort of thing because they have umpteen -> > thousand developers available. Postgres does not. -> -> Well, Oracle also started out, a long long time ago, on systems without -> unified buffer cache and so on, and so they *had* to write this stuff -> because otherwise data would not be cached. So Oracle can also afford to -> maintain it now because the code already exists. - -Well, actually, it isn't unified buffer cache that is the issue, but -rather the older SysV file system had pretty poor performance so -bypassing it was a bigger win that it is today. - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 853-3000 - + If your life is a hard drive, | 830 Blythe Avenue - + Christ can be your backup. | Drexel Hill, Pennsylvania 19026 - - - ----------------------------(end of broadcast)--------------------------- -TIP 4: Don't 'kill -9' the postmaster - - - -From pgsql-hackers-owner+M31893@postgresql.org Fri Nov 15 11:25:58 2002 -Return-path: -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id gAFHPvR10276 - for ; Fri, 15 Nov 2002 12:25:57 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP - id A2D5A4774A1; Fri, 15 Nov 2002 11:34:54 -0500 (EST) -Received: from postgresql.org (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with SMTP - id 5E898477132; Fri, 15 Nov 2002 11:15:45 -0500 (EST) -Received: from localhost (postgresql.org [64.49.215.8]) - by postgresql.org (Postfix) with ESMTP id 90CF1475B85 - for ; Mon, 11 Nov 2002 15:33:47 -0500 (EST) -Received: from Curtis-Vaio (unknown [63.164.0.45]) - by postgresql.org (Postfix) with SMTP id C6CB1475A3F - for ; Mon, 11 Nov 2002 15:33:46 -0500 (EST) -Received: from [127.0.0.1] by Curtis-Vaio - (ArGoSoft Mail Server Freeware, Version 1.8 (1.8.1.7)); Mon, 11 Nov 2002 16:33:42 -0400 -From: "Curtis Faith" -To: -Subject: [HACKERS] 500 tpsQL + WAL log implementation -Date: Mon, 11 Nov 2002 16:33:41 -0400 -Message-ID: -MIME-Version: 1.0 -Content-Type: text/plain; - charset="iso-8859-1" -Content-Transfer-Encoding: 7bit -X-Priority: 3 (Normal) -X-MSMail-Priority: Normal -X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) -X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 -Importance: Normal -X-Virus-Scanned: by AMaViS new-20020517 -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by AMaViS new-20020517 -Status: ROr - -I have been experimenting with empirical tests of file system and device -level writes to determine the actual constraints in order to speed up the WAL -logging code. - -Using a raw file partition and a time-based technique for determining the -optimal write position, I am able to get 8K writes physically written to disk -synchronously in the range of 500 to 650 writes per second using FreeBSD raw -device partitions on IDE disks (with write cache disabled). I will be -testing it soon under linux with 10,00RPM SCSI which should be even better. -It is my belief that the mechanism used to achieve these speeds could be -incorporated into the existing WAL logging code as an abstraction that looks -to the WAL code just like the file level access currently used. The current -speeds are limited by the speed of a single disk rotation. For a 7,200 RPM -disk this is 120/second, for a 10,000 RPM disk this is 166.66/second - -The mechanism works by adjusting the seek offset of the write by using -gettimeofday to determine approximately where the disk head is in its -rotation. The mechanism does not use any AIO calls. - -Assuming the following: - -1) Disk rotation time is 8.333ms or 8333us (7200 RPM). - -2) A write at offset 1,500K completes at system time 103s 000ms 000us - -3) A new write is requested at system time 103s 004ms 166us - -4) A 390K per rotation alignment of the data on the disk. - -5) A write must be sent at least 20K ahead of the current head position to -ensure that it is written in less than one rotation. - -It can be determined from the above that a write for an offset of something -slightly more than 195K past the last write, or offset 1,695K will be ahead -of the current location of the head and will therefore complete in less than -a single rotation's time. - -The disk specific metrics (rotation speed, bytes per rotation, base write -time, etc.) can be derived empirically through a tester program that would -take a few minutes to run and which could be run at log setup time. - -The obvious problem with the above mechanism is that the WAL log needs to be -able to read from the log file in transaction order during recovery. This -could be provided for using an abstraction that prepends the logical order -for each block written to the disk and makes sure that the log blocks contain -either a valid logical order number or some other marker indicating that the -block is not being used. - -A bitmap of blocks that have already been used would be kept in memory for -quickly determining the next set of possible unused blocks but this bitmap -would not need to be written to disk except during normal shutdown since in -the even of a failure the bitmaps would be reconstructed by reading all the -blocks from the disk. - -Checkpointing and something akin to log rotation could be handled using this -mechanism as well. - -So, MY REAL QUESTION is whether or not this is the sort of speed improvement -that warrants the work of writing the required abstraction layer and making -this very robust. The WAL code should remain essentially unchanged, with -perhaps new calls for the five or six routines used to access the log files, -and handle the equivalent of log rotation for raw device access. These new -calls would either use the current file based implementation or the new -logging mechanism depending on the configuration. - -I anticipate that the extra work required for a PostgreSQL administrator to -use the proposed logging mechanism would be to: - -1) Create a raw device partition of the appropriate size -2) Run the metrics tester for that device partition -3) Set the appropriate configuration parameters to indicate raw WAL logging - -I anticipate that the additional space requirements for this system would be -on the order of 10% to 15% beyond the current file-based implementation's -requirements. - -So, is this worth doing? Would a robust implementation likely be accepted for -7.4 assuming it can demonstrate speed improvements in the range of 500tps? - -- Curtis - - - - - - - - - - - - - - - - - ----------------------------(end of broadcast)--------------------------- -TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org - -On Sun, 2005-04-10 at 21:12 -0400, Bruce Momjian wrote: -> Jim C. Nasby wrote: -> > Maybe better for -hackers, but here it goes anyway... -> > -> > Has anyone looked at compressing WAL's before writing to disk? On a -> > system generating a lot of WAL it seems there might be some gains to be -> > had WAL data could be compressed before going to disk, since today's -> > machines are generally more I/O bound than CPU bound. And unlike the -> > base tables, you generally don't need to read the WAL, so you don't -> > really need to worry about not being able to quickly scan through the -> > data without decompressing it. -> -> I have never heard anyone talk about it, but it seems useful. I think -> compressing the page images written on first page modification since -> checkpoint would be a big win. - -Well it was discussed 2-3 years ago as part of the PITR preamble. You -may be surprised to read that over... - -A summary of thoughts to date on this are: - -xlog.c XLogInsert places backup blocks into the wal buffers before -insertion, so is the right place to do this. It would be possible to do -this before any LWlocks are taken, so would not not necessarily impair -scalability. - -Currently XLogInsert is a severe CPU bottleneck around the CRC -calculation, as identified recently by Tom. Digging further, the code -used seems to cause processor stalls on Intel CPUs, possibly responsible -for much of the CPU time. Discussions to move to a 32-bit CRC would also -be effected by this because of the byte-by-byte nature of the algorithm, -whatever the length of the generating polynomial. PostgreSQL's CRC -algorithm is the fastest BSD code available. Until improvement is made -there, I would not investigate compression further. Some input from -hardware tuning specialists is required... - -The current LZW compression code uses a 4096 byte lookback size, so that -would need to be modified to extend across a whole block. An -alternative, suggested originally by Tom and rediscovered by me because -I just don't read everybody's fine words in history, is to simply take -out the freespace in the middle of every heap block that consists of -zeros. - -Any solution in this area must take into account the variability of the -size of freespace in database blocks. Some databases have mostly full -blocks, others vary. There would also be considerable variation in -compressability of blocks, especially since some blocks (e.g. TOAST) are -likely to already be compressed. There'd need to be some testing done to -see exactly the point where the costs of compression produce realisable -benefits. - -So any solution must be able to cope with both compressed blocks and -non-compressed blocks. My current thinking is that this could be -achieved by using the spare fourth bit of the BkpBlocks portion of the -XLog structure, so that either all included BkpBlocks are compressed or -none of them are, and hope that allows benefit to shine through. Not -thought about heap/index issues. - -It is possible that an XLogWriter process could be used to assist in the -CRC and compression calculations also, an a similar process used to -assist decompression for recovery, in time. - -I regret I do not currently have time to pursue further. - -Best Regards, Simon Riggs - -From pgsql-hackers-owner+M65147=pgman=candle.pha.pa.us@postgresql.org Fri Mar 11 12:35:29 2005 -Return-path: -Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j2BIZSk26269 - for ; Fri, 11 Mar 2005 13:35:29 -0500 (EST) -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id 7C7DD545B4 - for ; Fri, 11 Mar 2005 18:35:25 +0000 (GMT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 31351-02 for ; - Fri, 11 Mar 2005 18:35:25 +0000 (GMT) -Received: from postgresql.org (svr1.postgresql.org [200.46.204.71]) - by svr1.postgresql.org (Postfix) with ESMTP id D316E545A2 - for ; Fri, 11 Mar 2005 18:35:24 +0000 (GMT) -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -Received: from localhost (unknown [200.46.204.144]) - by svr1.postgresql.org (Postfix) with ESMTP id E70B65640C - for ; Fri, 11 Mar 2005 18:33:21 +0000 (GMT) -Received: from svr1.postgresql.org ([200.46.204.71]) - by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) - with ESMTP id 29957-07 - for ; - Fri, 11 Mar 2005 18:33:19 +0000 (GMT) -Received: from hal.kabsi.at (bottom.kabsi.at [195.202.128.74]) - by svr1.postgresql.org (Postfix) with ESMTP id 9D9EE545EE - for ; Fri, 11 Mar 2005 18:33:17 +0000 (GMT) -Received: from [192.168.0.5] (h062040243020.plc.cm.kabsi.at [62.40.243.20]) - by hal.kabsi.at (8.11.1/) with ESMTP id j2BIX7a0000748810; - Fri, 11 Mar 2005 19:33:07 +0100 (CET) -Message-ID: <4231E416.4030900@cybertec.at> -Date: Fri, 11 Mar 2005 19:31:50 +0100 -From: =?ISO-8859-1?Q?Hans-J=FCrgen_Sch=F6nig?= -User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.3) Gecko/20040922 -X-Accept-Language: en-us, en -MIME-Version: 1.0 -To: Simon Riggs -cc: Tom Lane , - Mark Cave-Ayland , - pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Cost of XLogInsert CRC calculations -References: <9EB50F1A91413F4FA63019487FCD251D113169@WEBBASEDDC.webbasedltd.local> <23031.1110206390@sss.pgh.pa.us> <1110239639.6117.197.camel@localhost.localdomain> -In-Reply-To: <1110239639.6117.197.camel@localhost.localdomain> -Content-Type: text/plain; charset=us-ascii; format=flowed -Content-Transfer-Encoding: 7bit -X-Virus-Scanned: by amavisd-new at hub.org -X-Mailing-List: pgsql-hackers -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -X-Virus-Scanned: by amavisd-new at hub.org -Status: RO - - -> One of the things I was thinking about was whether we could use up those -> cycles more effectively. If we were to include a compression routine -> before we calculated the CRC that would -> - reduce the size of the blocks to be written, hence reduce size of xlog -> - reduce the following CRC calculation -> -> I was thinking about using a simple run-length encoding to massively -> shrink half-empty blocks with lots of zero padding, but we've already -> got code to LZW the data down also. -> -> Best Regards, Simon Riggs -> -> -> ---------------------------(end of broadcast)--------------------------- -> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org - - -Simon, - -I think having a compression routine in there could make real sense. -We have done some major I/O testing involving compression for a large -customer some time ago. We have seen that compressing / decompressing on -the fly is in MOST cases much faster than uncompressed I/O (try a simple -"cat file | ..." vs." zcat file.gz | ...") - the zcat version will be -faster on all platforms we have tried (Linux, AIX, Sun on some SAN -system, etc. ...). -Also, when building up a large database within one transaction the xlog -will eat a lot of storage - this can be quite annoying when you have to -deal with a lot of data). -Are there any technical reasons which would prevent somebody from -implementing compression? - - Best regards, - - Hans - --- -Cybertec Geschwinde u Schoenig -Schoengrabern 134, A-2020 Hollabrunn, Austria -Tel: +43/660/816 40 77 -www.cybertec.at, www.postgresql.at - - ----------------------------(end of broadcast)--------------------------- -TIP 5: Have you checked our extensive FAQ? - - http://www.postgresql.org/docs/faq - diff --git a/doc/TODO.detail/walcontrol b/doc/TODO.detail/walcontrol deleted file mode 100644 index f70bbf7b59..0000000000 --- a/doc/TODO.detail/walcontrol +++ /dev/null @@ -1,3379 +0,0 @@ -From pgsql-hackers-owner+M77861=pgman=candle.pha.pa.us@postgresql.org Fri Dec 23 05:19:20 2005 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -From: Simon Riggs -To: Stephen Frost -cc: Martijn van Oosterhout , - Jim C. Nasby , - bizgres-general , - pgsql-hackers@postgresql.org -In-Reply-To: <20051222223625.GC6026@ns.snowman.net> -References: <1135261893.2964.502.camel@localhost.localdomain> - <20051222183751.GG72143@pervasive.com> <20051222201826.GH21783@svana.org> - <1135289583.2964.536.camel@localhost.localdomain> - <20051222223625.GC6026@ns.snowman.net> -Date: Fri, 23 Dec 2005 10:18:43 +0000 -Message-ID: <1135333123.2964.589.camel@localhost.localdomain> -X-Mailer: Evolution 2.2.3 (2.2.3-2.fc4) -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.1 required=5 tests=[AWL=0.100] -X-Spam-Score: 0.1 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 4728 - -On Thu, 2005-12-22 at 17:36 -0500, Stephen Frost wrote: -> * Simon Riggs (simon@2ndquadrant.com) wrote: -> > On Thu, 2005-12-22 at 21:18 +0100, Martijn van Oosterhout wrote: -> > > Considering "WAL bypass" is code for "breaks PITR" -> > -> > No it isn't. All of the WAL bypass logic does *not* operate when PITR is -> > active. The WAL bypass logic is aimed at Data Warehouses, which -> > typically never operate in PITR mode for performance reasons, however -> > the choice is yours. - -OK, thanks for saying all of that; you probably speak for many in -raising these concerns. I'll answer each bit as we come to it. Suffice -to say, your concerns are good and so are the answers: - -> Eh? PITR mode is bad for performance? Maybe I missed something but I -> wouldn't have thought PITR would degrade regular performance all that -> badly. - -PITR mode is *not* bad for performance. On a very heavily loaded -write-intensive test system, the general PITR overhead on regular -performance was around 1% - so almost negligible. - -We have been discussing a number of optimizations to specific commands -that would allow them to avoid writing WAL and thus speed up their -performance. If archive_command is set then WAL will always be written; -if it is not set then these commands will (or could) go faster: - -- CREATE TABLE AS SELECT (in 8.1) -- COPY LOCK (patch submitted) -- COPY in same transaction as CREATE TABLE (patch submitted) -- INSERT SELECT in same transaction as CREATE TABLE (this discussion) - -(There are a number of other conditions also, such as there must be no -indexes on a table. All of which now documented with the patch) - -> So long as it doesn't take 15 minutes or some such to move the -> WAL to somewhere else (and I'm not sure that'd even slow things down..). -> For a Data Warehouse, have you got a better way of doing backups such -> that you don't lose at minimum most of a day's work? - -Yes. Don't just use the backup facilities on their own. Think about how -the architecture of your systems will work and see if there is a better -way when you look at very large systems. - -> I'm not exactly a -> big fan do doing a pg_dump every night either given that the database is -> 360GB. Much nicer to take a weekly dump of the database and then do -> PITR for a week or two before taking another dump of the db. - -e.g. Keep your reference data (low volume) in an Operational Data Store -(ODS) database, protected by archiving. Keep your main fact data (high -volume) in the Data Warehouse, but save the data in slices as you load -it, so that a recovery is simply a reload of the database: no PITR or -pg_dump required, so high performance data transformation and load work -is possible. This is a commonly used architectural design pattern. - -> I like the idea of making COPY go faster, but please don't break my -> backup system while you're at it. - -On a personal note, I would only add that I spent a long time working on -PITR and I would never design anything that would intentionally break it -(nor would patches be accepted that did that). That probably gives me -the confidence to approach designs that might look like I'm doing that, -but without actually straying over the edge. - -> I'm honestly kind of nervous about -> what you mean by checking it PITR is active- how is that done, exactly? -> Check if you have a script set to rotate the logs elsewhere? Or is it -> checking if you're in the taking-a-full-database-backup stage? Or what? - -Internally, we use XLogArchivingActive(). Externally this will be set -when the admin sets archive_command to a particular value. - -My original preference was for a parameter called archive_mode= ON | OFF -which would allow us to more easily discuss this, but this does not -currently exist. - -> What's the performance decrease when using PITR, and what's it from? Is -> it just that COPY isn't as fast? Honestly, I could live with COPY being -> not as fast as it could be if my backups work. :) - -These commands will not be optimized for speed when archive_command is set: -- CREATE TABLE AS SELECT (in 8.1) -- COPY LOCK (patch submitted) - -> Sorry for sounding concerned but, well, backups are very important and -> so is performance and I'm afraid either I've not read all the -> documentation about the issues being discussed here or there isn't -> enough out there to make sense of it all yet. :) - -If you choose PITR, then you are safe. If you do not, the crash recovery -of the database is not endangered by these optimizations. - -Hope that covers all of your concerns? - -I'm just writing a course that explains many of these techniques, -available in the New Year. - -Best Regards, Simon Riggs - - ----------------------------(end of broadcast)--------------------------- -TIP 2: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M78004=pgman=candle.pha.pa.us@postgresql.org Wed Dec 28 20:59:03 2005 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -From: Bruce Momjian -Message-ID: <200512290158.jBT1wEK28785@candle.pha.pa.us> -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -In-Reply-To: <20051226122206.GA12934@svana.org> -To: Martijn van Oosterhout -Date: Wed, 28 Dec 2005 20:58:14 -0500 (EST) -cc: Simon Riggs , Tom Lane , - Greg Stark , Rod Taylor , - Qingqing Zhou , pgsql-hackers@postgresql.org -X-Mailer: ELM [version 2.4ME+ PL121 (25)] -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.122 required=5 tests=[AWL=0.122] -X-Spam-Score: 0.122 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 3461 - - -Having read through this thread, I would like to propose a -syntax/behavior. - -I think we all now agree that the logging is more part of the table than -the command itself. Right now we have a COPY LOCK patch, but people are -going to want to control logging for INSERT INTO ... SELECT, and UPDATE, -and all sorts of other things, so I think we are best adding an ALTER -TABLE capability. I am thinking of this syntax: - - ALTER TABLE name RELIABILITY option - -where "option" is: - - DROP [ TABLE ON CRASH ] - DELETE [ ROWS ON CRASH ] - EXCLUSIVE - SHARE - -Let me explain each option. DROP would drop the table on a restart -after a non-clean shutdown. It would do _no_ logging on the table and -allow concurrent access, plus index access. DELETE is the same as DROP, -but it just truncates the table (perhaps TRUNCATE is a better word). - -EXCLUSIVE would allow only a single session to modify the table, and -would do all changes by appending to the table, similar to COPY LOCK. -EXCLUSIVE would also not allow indexes because those can not be isolated -like appending to the heap. EXCLUSIVE would write all dirty shared -buffers for the table and fsync them before committing. SHARE is the -functionality we have now, with full logging. - -Does this get us any closer to a TODO item? It isn't great, but I think -it is pretty clear, and I assume pg_dump would use ALTER to load each -table. The advanage is that the COPY statements themselves are -unchanged so they would work in loading into older versions of -PostgreSQL. - ---------------------------------------------------------------------------- - -Martijn van Oosterhout wrote: --- Start of PGP signed section. -> On Mon, Dec 26, 2005 at 12:03:27PM +0000, Simon Riggs wrote: -> > I would not be against such a table-level switch, but the exact -> > behaviour would need to be specified more closely before this became a -> > TODO item, IMHO. -> -> Well, I think at a per table level is the only sensible level. If a -> table isn't logged, neither are the indexes. After an unclean shutdown -> the data could be anywhere between OK and rubbish, with no way of -> finding out which way. -> -> > If someone has a 100 GB table, they would not appreciate the table being -> > truncated if a transaction to load 1 GB of data aborts, forcing recovery -> > of the 100 GB table. -> -> Ah, but wouldn't such a large table be partitioned in such a way that -> you could have the most recent partition having the loaded data. -> Personally, I think these "shared temp tables" have more applications -> than meet the eye. I've had systems with cache tables which could be -> wiped on boot. Though I think my preference would be to TRUNCATE rather -> than DROP on unclean shutdown. -> -> Have a nice day, -> -- -> Martijn van Oosterhout http://svana.org/kleptog/ -> > Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a -> > tool for doing 5% of the work and then sitting around waiting for someone -> > else to do the other 95% so you can sue them. --- End of PGP section, PGP failed! - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 359-1001 - + If your life is a hard drive, | 13 Roberts Road - + Christ can be your backup. | Newtown Square, Pennsylvania 19073 - ----------------------------(end of broadcast)--------------------------- -TIP 3: Have you checked our extensive FAQ? - - http://www.postgresql.org/docs/faq - -From pgsql-hackers-owner+M78007=pgman=candle.pha.pa.us@postgresql.org Wed Dec 28 22:06:13 2005 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -Message-ID: <43B3527A.4040709@commandprompt.com> -Date: Wed, 28 Dec 2005 19:05:30 -0800 -From: Joshua D. Drake -Organization: Command Prompt, Inc. -User-Agent: Mozilla Thunderbird 1.0.2 (Windows/20050317) -X-Accept-Language: en-us, en -To: Bruce Momjian -cc: Martijn van Oosterhout , - Simon Riggs , Tom Lane , - Greg Stark , Rod Taylor , - Qingqing Zhou , pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -References: <200512290158.jBT1wEK28785@candle.pha.pa.us> -In-Reply-To: <200512290158.jBT1wEK28785@candle.pha.pa.us> -X-Greylist: Sender succeded SMTP AUTH authentication, not delayed by milter-greylist-1.6 (hosting.commandprompt.com [192.168.1.101]); Wed, 28 Dec 2005 18:57:25 -0800 (PST) -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.05 required=5 tests=[AWL=0.050, UPPERCASE_25_50=0] -X-Spam-Score: 0.05 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 725 - - now agree that the logging is more part of the table than -> the command itself. Right now we have a COPY LOCK patch, but people are -> going to want to control logging for INSERT INTO ... SELECT, and UPDATE, -> and all sorts of other things, so I think we are best adding an ALTER -> TABLE capability. I am thinking of this syntax: -> -> ALTER TABLE name RELIABILITY option -> -> where "option" is: -> -> DROP [ TABLE ON CRASH ] -> DELETE [ ROWS ON CRASH ] -> EXCLUSIVE -> SHARE - -I would say ON FAILURE (Crash just seems way to scary :)) - -Joshua D. Drake - - ----------------------------(end of broadcast)--------------------------- -TIP 3: Have you checked our extensive FAQ? - - http://www.postgresql.org/docs/faq - -From pgsql-hackers-owner+M78008=pgman=candle.pha.pa.us@postgresql.org Wed Dec 28 23:09:58 2005 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -From: Bruce Momjian -Message-ID: <200512290409.jBT49LD13611@candle.pha.pa.us> -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -In-Reply-To: <43B3527A.4040709@commandprompt.com> -To: Joshua D. Drake -Date: Wed, 28 Dec 2005 23:09:21 -0500 (EST) -cc: Martijn van Oosterhout , - Simon Riggs , Tom Lane , - Greg Stark , Rod Taylor , - Qingqing Zhou , pgsql-hackers@postgresql.org -X-Mailer: ELM [version 2.4ME+ PL121 (25)] -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.122 required=5 tests=[AWL=0.122, UPPERCASE_25_50=0] -X-Spam-Score: 0.122 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 1111 - -Joshua D. Drake wrote: -> now agree that the logging is more part of the table than -> > the command itself. Right now we have a COPY LOCK patch, but people are -> > going to want to control logging for INSERT INTO ... SELECT, and UPDATE, -> > and all sorts of other things, so I think we are best adding an ALTER -> > TABLE capability. I am thinking of this syntax: -> > -> > ALTER TABLE name RELIABILITY option -> > -> > where "option" is: -> > -> > DROP [ TABLE ON CRASH ] -> > DELETE [ ROWS ON CRASH ] -> > EXCLUSIVE -> > SHARE -> -> I would say ON FAILURE (Crash just seems way to scary :)) - -Agreed, maybe ON RECOVERY. - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 359-1001 - + If your life is a hard drive, | 13 Roberts Road - + Christ can be your backup. | Newtown Square, Pennsylvania 19073 - ----------------------------(end of broadcast)--------------------------- -TIP 9: In versions below 8.0, the planner will ignore your desire to - choose an index scan if your joining column's datatypes do not - match - -From simon@2ndquadrant.com Thu Dec 29 08:19:47 2005 -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -From: Simon Riggs -To: Bruce Momjian -cc: Martijn van Oosterhout , Tom Lane , - Greg Stark , Rod Taylor , - Qingqing Zhou , pgsql-hackers@postgresql.org -In-Reply-To: <200512290158.jBT1wEK28785@candle.pha.pa.us> -References: <200512290158.jBT1wEK28785@candle.pha.pa.us> -Date: Thu, 29 Dec 2005 13:19:45 +0000 -Message-ID: <1135862385.2964.804.camel@localhost.localdomain> -X-Mailer: Evolution 2.2.3 (2.2.3-2.fc4) -Content-Length: 7026 - -On Wed, 2005-12-28 at 20:58 -0500, Bruce Momjian wrote: -> Having read through this thread, I would like to propose a -> syntax/behavior. -> -> I think we all now agree that the logging is more part of the table than -> the command itself. Right now we have a COPY LOCK patch, but people are -> going to want to control logging for INSERT INTO ... SELECT, and UPDATE, -> and all sorts of other things, so I think we are best adding an ALTER -> TABLE capability. I am thinking of this syntax: -> -> ALTER TABLE name RELIABILITY option -> -> where "option" is: -> -> DROP [ TABLE ON CRASH ] -> DELETE [ ROWS ON CRASH ] -> EXCLUSIVE -> SHARE -> -> Let me explain each option. DROP would drop the table on a restart -> after a non-clean shutdown. It would do _no_ logging on the table and -> allow concurrent access, plus index access. DELETE is the same as DROP, -> but it just truncates the table (perhaps TRUNCATE is a better word). -> -> EXCLUSIVE would allow only a single session to modify the table, and -> would do all changes by appending to the table, similar to COPY LOCK. -> EXCLUSIVE would also not allow indexes because those can not be isolated -> like appending to the heap. EXCLUSIVE would write all dirty shared -> buffers for the table and fsync them before committing. SHARE is the -> functionality we have now, with full logging. -> -> Does this get us any closer to a TODO item? It isn't great, but I think -> it is pretty clear, and I assume pg_dump would use ALTER to load each -> table. The advanage is that the COPY statements themselves are -> unchanged so they would work in loading into older versions of -> PostgreSQL. - -First off, thanks for summarising a complex thread. - -My view would be that this thread has been complex because everybody has -expressed a somewhat different requirement, which could be broken down -as: -1. The need for a multi-user-accessible yet temporary table -2. Loading data into a table immediately after it is created (i.e. in -same transaction), including but not limited to a reload from pg_dump -3. How to load data quickly into an existing table (COPY) -4. How to add/modify data quickly in an existing table (INSERT SELECT, -UPDATE) - -I can see the need for all of those individually; my existing patch -submission covers (2) and (3) only. I very much like your thought to -coalesce these various requirements into a single coherent model. - -For requirement (1), table level options make sense. We would: -- CREATE TABLE ALLTHINGS -- ALTER TABLE ALLTHINGS RELIABILITY DELETE ROWS ON RECOVERY -- lots of SQL, all fast because not logged - -(2) is catered for adequately by the existing COPY patch i.e. it will -detect whether a table has just been created and then avoid writing WAL. -In the patch, pg_dump has *not* been altered to use COPY LOCK, so a -pg_dump *will* work with any other version of PostgreSQL, which *would -not* be the case if we added ALTER TABLE ... RELIABILITY statements into -it. Also, a pg_dump created at an earlier version could also be loaded -faster using the patch. The only requirement is to issue all SQL as part -of the same transaction - which is catered for by the ---single-transaction option on pg_restore and psql. So (2) is catered -for fully without the need for an ALTER TABLE ... RELIABILITY statement -or COPY LOCK. - -For requirement (3), I would use table level options like this: -(the table already exists and is reasonably big; we should not assume -that everybody can and does use partitioning) -- ALTER TABLE RELIABILITY ALLTHINGS2 EXCLUSIVE -- COPY -- ALTER TABLE RELIABILITY ALLTHINGS2 SHARE - -For a load into an existing table I would always do all three actions -together. COPY LOCK does exactly that *and* does it atomically. - -The two ways of doing (3) have a few pros/cons either way: -Pro for ALTER TABLE: -- same syntax as req (1) -- doesn't need the keyword LOCK -- allows INSERT SELECT, UPDATE operations also (req 4) -Cons: -- existing programs have to add additional statements to take advantage -of this; with COPY LOCK we would add just a single keyword -- operation is not atomic, which might lead to some operations waiting -for a lock to operate as unlogged, since they would execute before the -second ALTER TABLE gets there -- operation will be understood by some, but not others. They will forget -to switch the RELIABILITY back on and then lose their whole table when -the database crashes. (watch...) - -...but would it be a problem to have both? - - -So, my thinking would be to separate things into two: -a) Add a TODO item "shared temp tables" that caters for (1) and (4) - - ALTER TABLE name RELIABILITY - {DELETE ROWS AT RECOVERY | FULL RECOVERY} -(syntax TBD) - -which would -- truncate all rows and remove all index entries during recovery -- use shared_buffers, not temp_buffers -- never write xlog records, even when in PITR mode -- would avoid writing WAL for both heap *and* index tuples - -b) Leave the COPY patch as is, since it caters for reqs (2) and (3) as -*separate* optimizations (but using a common infrastructure in code). -[This work was based upon discussions on -hackers only 6 months ago, so -its not like its been snuck in or anything -http://archives.postgresql.org/pgsql-hackers/2005-06/msg00069.php -http://archives.postgresql.org/pgsql-hackers/2005-06/msg00075.php ] - -These two thoughts are separable. There is no need to -have-both-or-neither within PostgreSQL. - -Eventually, I'd like all of these options, as a database designer. - -Best Regards, Simon Riggs - -> -------------------------------------------------------------------------- -> -> Martijn van Oosterhout wrote: -> -- Start of PGP signed section. -> > On Mon, Dec 26, 2005 at 12:03:27PM +0000, Simon Riggs wrote: -> > > I would not be against such a table-level switch, but the exact -> > > behaviour would need to be specified more closely before this became a -> > > TODO item, IMHO. -> > -> > Well, I think at a per table level is the only sensible level. If a -> > table isn't logged, neither are the indexes. After an unclean shutdown -> > the data could be anywhere between OK and rubbish, with no way of -> > finding out which way. -> > -> > > If someone has a 100 GB table, they would not appreciate the table being -> > > truncated if a transaction to load 1 GB of data aborts, forcing recovery -> > > of the 100 GB table. -> > -> > Ah, but wouldn't such a large table be partitioned in such a way that -> > you could have the most recent partition having the loaded data. -> > Personally, I think these "shared temp tables" have more applications -> > than meet the eye. I've had systems with cache tables which could be -> > wiped on boot. Though I think my preference would be to TRUNCATE rather -> > than DROP on unclean shutdown. -> > -> > Have a nice day, -> > -- -> > Martijn van Oosterhout http://svana.org/kleptog/ -> > > Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a -> > > tool for doing 5% of the work and then sitting around waiting for someone -> > > else to do the other 95% so you can sue them. -> -- End of PGP section, PGP failed! -> - -From pgsql-hackers-owner+M78019=pgman=candle.pha.pa.us@postgresql.org Thu Dec 29 08:20:11 2005 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -From: Simon Riggs -To: Bruce Momjian -cc: Martijn van Oosterhout , Tom Lane , - Greg Stark , Rod Taylor , - Qingqing Zhou , pgsql-hackers@postgresql.org -In-Reply-To: <200512290158.jBT1wEK28785@candle.pha.pa.us> -References: <200512290158.jBT1wEK28785@candle.pha.pa.us> -Date: Thu, 29 Dec 2005 13:19:45 +0000 -Message-ID: <1135862385.2964.804.camel@localhost.localdomain> -X-Mailer: Evolution 2.2.3 (2.2.3-2.fc4) -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.112 required=5 tests=[AWL=0.112] -X-Spam-Score: 0.112 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 7139 - -On Wed, 2005-12-28 at 20:58 -0500, Bruce Momjian wrote: -> Having read through this thread, I would like to propose a -> syntax/behavior. -> -> I think we all now agree that the logging is more part of the table than -> the command itself. Right now we have a COPY LOCK patch, but people are -> going to want to control logging for INSERT INTO ... SELECT, and UPDATE, -> and all sorts of other things, so I think we are best adding an ALTER -> TABLE capability. I am thinking of this syntax: -> -> ALTER TABLE name RELIABILITY option -> -> where "option" is: -> -> DROP [ TABLE ON CRASH ] -> DELETE [ ROWS ON CRASH ] -> EXCLUSIVE -> SHARE -> -> Let me explain each option. DROP would drop the table on a restart -> after a non-clean shutdown. It would do _no_ logging on the table and -> allow concurrent access, plus index access. DELETE is the same as DROP, -> but it just truncates the table (perhaps TRUNCATE is a better word). -> -> EXCLUSIVE would allow only a single session to modify the table, and -> would do all changes by appending to the table, similar to COPY LOCK. -> EXCLUSIVE would also not allow indexes because those can not be isolated -> like appending to the heap. EXCLUSIVE would write all dirty shared -> buffers for the table and fsync them before committing. SHARE is the -> functionality we have now, with full logging. -> -> Does this get us any closer to a TODO item? It isn't great, but I think -> it is pretty clear, and I assume pg_dump would use ALTER to load each -> table. The advanage is that the COPY statements themselves are -> unchanged so they would work in loading into older versions of -> PostgreSQL. - -First off, thanks for summarising a complex thread. - -My view would be that this thread has been complex because everybody has -expressed a somewhat different requirement, which could be broken down -as: -1. The need for a multi-user-accessible yet temporary table -2. Loading data into a table immediately after it is created (i.e. in -same transaction), including but not limited to a reload from pg_dump -3. How to load data quickly into an existing table (COPY) -4. How to add/modify data quickly in an existing table (INSERT SELECT, -UPDATE) - -I can see the need for all of those individually; my existing patch -submission covers (2) and (3) only. I very much like your thought to -coalesce these various requirements into a single coherent model. - -For requirement (1), table level options make sense. We would: -- CREATE TABLE ALLTHINGS -- ALTER TABLE ALLTHINGS RELIABILITY DELETE ROWS ON RECOVERY -- lots of SQL, all fast because not logged - -(2) is catered for adequately by the existing COPY patch i.e. it will -detect whether a table has just been created and then avoid writing WAL. -In the patch, pg_dump has *not* been altered to use COPY LOCK, so a -pg_dump *will* work with any other version of PostgreSQL, which *would -not* be the case if we added ALTER TABLE ... RELIABILITY statements into -it. Also, a pg_dump created at an earlier version could also be loaded -faster using the patch. The only requirement is to issue all SQL as part -of the same transaction - which is catered for by the ---single-transaction option on pg_restore and psql. So (2) is catered -for fully without the need for an ALTER TABLE ... RELIABILITY statement -or COPY LOCK. - -For requirement (3), I would use table level options like this: -(the table already exists and is reasonably big; we should not assume -that everybody can and does use partitioning) -- ALTER TABLE RELIABILITY ALLTHINGS2 EXCLUSIVE -- COPY -- ALTER TABLE RELIABILITY ALLTHINGS2 SHARE - -For a load into an existing table I would always do all three actions -together. COPY LOCK does exactly that *and* does it atomically. - -The two ways of doing (3) have a few pros/cons either way: -Pro for ALTER TABLE: -- same syntax as req (1) -- doesn't need the keyword LOCK -- allows INSERT SELECT, UPDATE operations also (req 4) -Cons: -- existing programs have to add additional statements to take advantage -of this; with COPY LOCK we would add just a single keyword -- operation is not atomic, which might lead to some operations waiting -for a lock to operate as unlogged, since they would execute before the -second ALTER TABLE gets there -- operation will be understood by some, but not others. They will forget -to switch the RELIABILITY back on and then lose their whole table when -the database crashes. (watch...) - -...but would it be a problem to have both? - - -So, my thinking would be to separate things into two: -a) Add a TODO item "shared temp tables" that caters for (1) and (4) - - ALTER TABLE name RELIABILITY - {DELETE ROWS AT RECOVERY | FULL RECOVERY} -(syntax TBD) - -which would -- truncate all rows and remove all index entries during recovery -- use shared_buffers, not temp_buffers -- never write xlog records, even when in PITR mode -- would avoid writing WAL for both heap *and* index tuples - -b) Leave the COPY patch as is, since it caters for reqs (2) and (3) as -*separate* optimizations (but using a common infrastructure in code). -[This work was based upon discussions on -hackers only 6 months ago, so -its not like its been snuck in or anything -http://archives.postgresql.org/pgsql-hackers/2005-06/msg00069.php -http://archives.postgresql.org/pgsql-hackers/2005-06/msg00075.php ] - -These two thoughts are separable. There is no need to -have-both-or-neither within PostgreSQL. - -Eventually, I'd like all of these options, as a database designer. - -Best Regards, Simon Riggs - -> -------------------------------------------------------------------------- -> -> Martijn van Oosterhout wrote: -> -- Start of PGP signed section. -> > On Mon, Dec 26, 2005 at 12:03:27PM +0000, Simon Riggs wrote: -> > > I would not be against such a table-level switch, but the exact -> > > behaviour would need to be specified more closely before this became a -> > > TODO item, IMHO. -> > -> > Well, I think at a per table level is the only sensible level. If a -> > table isn't logged, neither are the indexes. After an unclean shutdown -> > the data could be anywhere between OK and rubbish, with no way of -> > finding out which way. -> > -> > > If someone has a 100 GB table, they would not appreciate the table being -> > > truncated if a transaction to load 1 GB of data aborts, forcing recovery -> > > of the 100 GB table. -> > -> > Ah, but wouldn't such a large table be partitioned in such a way that -> > you could have the most recent partition having the loaded data. -> > Personally, I think these "shared temp tables" have more applications -> > than meet the eye. I've had systems with cache tables which could be -> > wiped on boot. Though I think my preference would be to TRUNCATE rather -> > than DROP on unclean shutdown. -> > -> > Have a nice day, -> > -- -> > Martijn van Oosterhout http://svana.org/kleptog/ -> > > Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a -> > > tool for doing 5% of the work and then sitting around waiting for someone -> > > else to do the other 95% so you can sue them. -> -- End of PGP section, PGP failed! -> - - ----------------------------(end of broadcast)--------------------------- -TIP 6: explain analyze is your friend - -From pgsql-hackers-owner+M78021=pgman=candle.pha.pa.us@postgresql.org Thu Dec 29 09:35:58 2005 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -From: Rod Taylor -To: Simon Riggs -cc: Bruce Momjian , - Martijn van Oosterhout , Tom Lane , - Greg Stark , Qingqing Zhou , - pgsql-hackers@postgresql.org -In-Reply-To: <1135862385.2964.804.camel@localhost.localdomain> -References: <200512290158.jBT1wEK28785@candle.pha.pa.us> - <1135862385.2964.804.camel@localhost.localdomain> -Date: Thu, 29 Dec 2005 09:35:27 -0500 -Message-ID: <1135866927.61038.13.camel@home> -X-Mailer: Evolution 2.4.2.1 FreeBSD GNOME Team Port -X-SA-Exim-Mail-From: pg@rbt.ca -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -X-SA-Exim-Version: 3.1 (built Tue Feb 24 05:09:27 GMT 2004) -X-SA-Exim-Scanned: Yes -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.024 required=5 tests=[AWL=0.024, UPPERCASE_25_50=0] -X-Spam-Score: 0.024 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 506 - - -> So, my thinking would be to separate things into two: -> a) Add a TODO item "shared temp tables" that caters for (1) and (4) -> -> ALTER TABLE name RELIABILITY -> {DELETE ROWS AT RECOVERY | FULL RECOVERY} -> (syntax TBD) - -DELETE ROWS AT RECOVERY would need to be careful or disallowed when -referenced via a foreign key to ensure the database is not restored in -an inconsistent state. - --- - - ----------------------------(end of broadcast)--------------------------- -TIP 2: Don't 'kill -9' the postmaster - -From pg@rbt.ca Thu Dec 29 09:35:35 2005 -From: Rod Taylor -To: Simon Riggs -cc: Bruce Momjian , - Martijn van Oosterhout , Tom Lane , - Greg Stark , Qingqing Zhou , - pgsql-hackers@postgresql.org -In-Reply-To: <1135862385.2964.804.camel@localhost.localdomain> -References: <200512290158.jBT1wEK28785@candle.pha.pa.us> - <1135862385.2964.804.camel@localhost.localdomain> -Date: Thu, 29 Dec 2005 09:35:27 -0500 -Message-ID: <1135866927.61038.13.camel@home> -X-Mailer: Evolution 2.4.2.1 FreeBSD GNOME Team Port -X-SA-Exim-Mail-From: pg@rbt.ca -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on psi.look.ca -X-Spam-Level: -X-Spam-Status: No, hits=0.7 required=9.0 tests=UPPERCASE_25_50 autolearn=no - version=2.63 -X-SA-Exim-Version: 3.1 (built Tue Feb 24 05:09:27 GMT 2004) -X-SA-Exim-Scanned: Yes -Content-Length: 393 - - -> So, my thinking would be to separate things into two: -> a) Add a TODO item "shared temp tables" that caters for (1) and (4) -> -> ALTER TABLE name RELIABILITY -> {DELETE ROWS AT RECOVERY | FULL RECOVERY} -> (syntax TBD) - -DELETE ROWS AT RECOVERY would need to be careful or disallowed when -referenced via a foreign key to ensure the database is not restored in -an inconsistent state. - --- - -From pgsql-hackers-owner+M78022=pgman=candle.pha.pa.us@postgresql.org Thu Dec 29 10:10:57 2005 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -From: Simon Riggs -To: Rod Taylor -cc: Bruce Momjian , - Martijn van Oosterhout , Tom Lane , - Greg Stark , Qingqing Zhou , - pgsql-hackers@postgresql.org -In-Reply-To: <1135866927.61038.13.camel@home> -References: <200512290158.jBT1wEK28785@candle.pha.pa.us> - <1135862385.2964.804.camel@localhost.localdomain> - <1135866927.61038.13.camel@home> -Date: Thu, 29 Dec 2005 15:10:40 +0000 -Message-ID: <1135869040.2964.824.camel@localhost.localdomain> -X-Mailer: Evolution 2.2.3 (2.2.3-2.fc4) -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.113 required=5 tests=[AWL=0.113] -X-Spam-Score: 0.113 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 888 - -On Thu, 2005-12-29 at 09:35 -0500, Rod Taylor wrote: -> > So, my thinking would be to separate things into two: -> > a) Add a TODO item "shared temp tables" that caters for (1) and (4) -> > -> > ALTER TABLE name RELIABILITY -> > {DELETE ROWS AT RECOVERY | FULL RECOVERY} -> > (syntax TBD) -> -> DELETE ROWS AT RECOVERY would need to be careful or disallowed when -> referenced via a foreign key to ensure the database is not restored in -> an inconsistent state. - -I think we'd need to apply the same rule as we do for temp tables: they -cannot be referenced by a permanent table. - -There are possibly some other restrictions also. Anyone? - -Best Regards, Simon Riggs - - ----------------------------(end of broadcast)--------------------------- -TIP 9: In versions below 8.0, the planner will ignore your desire to - choose an index scan if your joining column's datatypes do not - match - -From tgl@sss.pgh.pa.us Thu Dec 29 11:12:13 2005 -To: Simon Riggs -cc: Bruce Momjian , - Martijn van Oosterhout , Greg Stark , - Rod Taylor , Qingqing Zhou , - pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -In-Reply-To: <1135862385.2964.804.camel@localhost.localdomain> -References: <200512290158.jBT1wEK28785@candle.pha.pa.us> <1135862385.2964.804.camel@localhost.localdomain> -Comments: In-reply-to Simon Riggs - message dated "Thu, 29 Dec 2005 13:19:45 +0000" -Date: Thu, 29 Dec 2005 11:12:11 -0500 -Message-ID: <7273.1135872731@sss.pgh.pa.us> -From: Tom Lane -Content-Length: 1963 - -Simon Riggs writes: -> My view would be that this thread has been complex because everybody has -> expressed a somewhat different requirement, which could be broken down -> as: -> 1. The need for a multi-user-accessible yet temporary table -> 2. Loading data into a table immediately after it is created (i.e. in -> same transaction), including but not limited to a reload from pg_dump -> 3. How to load data quickly into an existing table (COPY) -> 4. How to add/modify data quickly in an existing table (INSERT SELECT, -> UPDATE) - -> I can see the need for all of those individually; my existing patch -> submission covers (2) and (3) only. I very much like your thought to -> coalesce these various requirements into a single coherent model. - -However, you then seem to be arguing for still using the COPY LOCK -syntax, which I think Bruce intended would go away in favor of using -these ALTER commands. Certainly that's what I'd prefer --- COPY has -got too darn many options already. - -> In the patch, pg_dump has *not* been altered to use COPY LOCK, so a -> pg_dump *will* work with any other version of PostgreSQL, which *would -> not* be the case if we added ALTER TABLE ... RELIABILITY statements into -> it. - -Wrong --- the good thing about ALTER TABLE is that an old version of -Postgres would simply reject it and keep going. Therefore we could get -the speedup in dumps without losing compatibility, which is not true -of COPY LOCK. - -BTW, this is a perfect example of the use-case for not abandoning a -dump-file load simply because one command fails. (We have relied on -this sort of reasoning many times before, too, for example by using -"SET default_with_oids" in preference to CREATE TABLE WITH/WITHOUT OIDS.) -I don't think that "wrap the whole load into begin/end" is really a very -workable answer, because there are far too many scenarios where you -can't do that. Another one where it doesn't help is a data-only dump. - - regards, tom lane - -From pgsql-hackers-owner+M78028=pgman=candle.pha.pa.us@postgresql.org Thu Dec 29 11:12:41 2005 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -To: Simon Riggs -cc: Bruce Momjian , - Martijn van Oosterhout , Greg Stark , - Rod Taylor , Qingqing Zhou , - pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -In-Reply-To: <1135862385.2964.804.camel@localhost.localdomain> -References: <200512290158.jBT1wEK28785@candle.pha.pa.us> <1135862385.2964.804.camel@localhost.localdomain> -Comments: In-reply-to Simon Riggs - message dated "Thu, 29 Dec 2005 13:19:45 +0000" -Date: Thu, 29 Dec 2005 11:12:11 -0500 -Message-ID: <7273.1135872731@sss.pgh.pa.us> -From: Tom Lane -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.053 required=5 tests=[AWL=0.053] -X-Spam-Score: 0.053 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 2075 - -Simon Riggs writes: -> My view would be that this thread has been complex because everybody has -> expressed a somewhat different requirement, which could be broken down -> as: -> 1. The need for a multi-user-accessible yet temporary table -> 2. Loading data into a table immediately after it is created (i.e. in -> same transaction), including but not limited to a reload from pg_dump -> 3. How to load data quickly into an existing table (COPY) -> 4. How to add/modify data quickly in an existing table (INSERT SELECT, -> UPDATE) - -> I can see the need for all of those individually; my existing patch -> submission covers (2) and (3) only. I very much like your thought to -> coalesce these various requirements into a single coherent model. - -However, you then seem to be arguing for still using the COPY LOCK -syntax, which I think Bruce intended would go away in favor of using -these ALTER commands. Certainly that's what I'd prefer --- COPY has -got too darn many options already. - -> In the patch, pg_dump has *not* been altered to use COPY LOCK, so a -> pg_dump *will* work with any other version of PostgreSQL, which *would -> not* be the case if we added ALTER TABLE ... RELIABILITY statements into -> it. - -Wrong --- the good thing about ALTER TABLE is that an old version of -Postgres would simply reject it and keep going. Therefore we could get -the speedup in dumps without losing compatibility, which is not true -of COPY LOCK. - -BTW, this is a perfect example of the use-case for not abandoning a -dump-file load simply because one command fails. (We have relied on -this sort of reasoning many times before, too, for example by using -"SET default_with_oids" in preference to CREATE TABLE WITH/WITHOUT OIDS.) -I don't think that "wrap the whole load into begin/end" is really a very -workable answer, because there are far too many scenarios where you -can't do that. Another one where it doesn't help is a data-only dump. - - regards, tom lane - ----------------------------(end of broadcast)--------------------------- -TIP 2: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M78025=pgman=candle.pha.pa.us@postgresql.org Thu Dec 29 10:57:46 2005 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -Message-ID: <51082.68.143.134.146.1135872877.squirrel@www.dunslane.net> -Date: Thu, 29 Dec 2005 10:14:37 -0600 (CST) -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -From: Andrew Dunstan -To: -In-Reply-To: <200512290158.jBT1wEK28785@candle.pha.pa.us> -References: <200512290158.jBT1wEK28785@candle.pha.pa.us> -X-Priority: 3 -Importance: Normal -X-MSMail-Priority: Normal -cc: , , , - , , , - -X-Mailer: SquirrelMail (version 1.2.5) -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.082 required=5 tests=[AWL=0.082] -X-Spam-Score: 0.082 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 1185 - -Bruce Momjian said: -> DROP would drop the table on a restart -> after a non-clean shutdown. It would do _no_ logging on the table and -> allow concurrent access, plus index access. DELETE is the same as -> DROP, but it just truncates the table (perhaps TRUNCATE is a better -> word). -> -> EXCLUSIVE would allow only a single session to modify the table, and -> would do all changes by appending to the table, similar to COPY LOCK. -> EXCLUSIVE would also not allow indexes because those can not be -> isolated like appending to the heap. EXCLUSIVE would write all dirty -> shared buffers for the table and fsync them before committing. SHARE -> is the functionality we have now, with full logging. - - -I an horribly scared that this will be used as a "performance boost" for -normal use. I would at least like to see some restrictions that make it -harder to mis-use. Perhaps restrict to superuser? - -cheers - -andrew - - - - - ----------------------------(end of broadcast)--------------------------- -TIP 1: if posting/reading through Usenet, please send an appropriate - subscribe-nomail command to majordomo@postgresql.org so that your - message can get through to the mailing list cleanly - -From tgl@sss.pgh.pa.us Thu Dec 29 11:24:30 2005 -To: Bruce Momjian -cc: Andrew Dunstan , kleptog@svana.org, - simon@2ndquadrant.com, gsstark@mit.edu, pg@rbt.ca, zhouqq@cs.toronto.edu, - pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -In-Reply-To: <200512291605.jBTG5gi00396@candle.pha.pa.us> -References: <200512291605.jBTG5gi00396@candle.pha.pa.us> -Comments: In-reply-to Bruce Momjian - message dated "Thu, 29 Dec 2005 11:05:42 -0500" -Date: Thu, 29 Dec 2005 11:24:28 -0500 -Message-ID: <7966.1135873468@sss.pgh.pa.us> -From: Tom Lane -Content-Length: 612 - -Bruce Momjian writes: -> Andrew Dunstan wrote: ->> I an horribly scared that this will be used as a "performance boost" for ->> normal use. I would at least like to see some restrictions that make it ->> harder to mis-use. Perhaps restrict to superuser? - -> Certainly restrict to table owner. - -I can see the argument for superuser-only: decisions about data -integrity tradeoffs should be reserved to the DBA, who is the one who -will get blamed if the database loses data, no matter how stupid his -users are. - -But I'm not wedded to that. I could live with table-owner. - - regards, tom lane - -From pgsql-hackers-owner+M78031=pgman=candle.pha.pa.us@postgresql.org Thu Dec 29 11:38:17 2005 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -From: Bruce Momjian -Message-ID: <200512291637.jBTGbdC03848@candle.pha.pa.us> -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -In-Reply-To: <7273.1135872731@sss.pgh.pa.us> -To: Tom Lane -Date: Thu, 29 Dec 2005 11:37:39 -0500 (EST) -cc: Simon Riggs , - Martijn van Oosterhout , Greg Stark , - Rod Taylor , Qingqing Zhou , - pgsql-hackers@postgresql.org -X-Mailer: ELM [version 2.4ME+ PL121 (25)] -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.122 required=5 tests=[AWL=0.122] -X-Spam-Score: 0.122 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 3932 - -Tom Lane wrote: -> Simon Riggs writes: -> > My view would be that this thread has been complex because everybody has -> > expressed a somewhat different requirement, which could be broken down -> > as: -> > 1. The need for a multi-user-accessible yet temporary table -> > 2. Loading data into a table immediately after it is created (i.e. in -> > same transaction), including but not limited to a reload from pg_dump -> > 3. How to load data quickly into an existing table (COPY) -> > 4. How to add/modify data quickly in an existing table (INSERT SELECT, -> > UPDATE) -> -> > I can see the need for all of those individually; my existing patch -> > submission covers (2) and (3) only. I very much like your thought to -> > coalesce these various requirements into a single coherent model. -> -> However, you then seem to be arguing for still using the COPY LOCK -> syntax, which I think Bruce intended would go away in favor of using -> these ALTER commands. Certainly that's what I'd prefer --- COPY has -> got too darn many options already. -> -> > In the patch, pg_dump has *not* been altered to use COPY LOCK, so a -> > pg_dump *will* work with any other version of PostgreSQL, which *would -> > not* be the case if we added ALTER TABLE ... RELIABILITY statements into -> > it. -> -> Wrong --- the good thing about ALTER TABLE is that an old version of -> Postgres would simply reject it and keep going. Therefore we could get -> the speedup in dumps without losing compatibility, which is not true -> of COPY LOCK. -> -> BTW, this is a perfect example of the use-case for not abandoning a -> dump-file load simply because one command fails. (We have relied on -> this sort of reasoning many times before, too, for example by using -> "SET default_with_oids" in preference to CREATE TABLE WITH/WITHOUT OIDS.) -> I don't think that "wrap the whole load into begin/end" is really a very -> workable answer, because there are far too many scenarios where you -> can't do that. Another one where it doesn't help is a data-only dump. - -Yep, Tom is echoing my reaction. There is a temptation to add things up -onto existing commands, e.g. LOCK, and while it works, it makes for some -very complex user API's. Having COPY behave differently because it is -in a transaction is fine as long as it is user-invisible, but once you -require users to do that to get the speedup, it isn't user-invisible -anymore. - -(I can see it now, "Why is pg_dump putting things in transactions?", -"Because it prevents it from being logged." "Oh, should I be doing that -in my code?" "Perhaps, if you want ..." You can see where that -discussion is going. Having them see "ATER TABLE ... RELIBILITY -TRUNCATE" is very clear, and very clear on how it can be used in user -code.) - -I think there is great utility in giving users one API, namely -RELIABILITY (or some other keyword), and telling them that is where they -control logging. I realize adding one keyword, LOCK, to an existing -command isn't a big deal, but once you decentralize your API enough -times, you end up with a terribly complex database system. It is this -design rigidity that helps make PostgreSQL so much easier to use than -other database systems. - -I do think it is valid concern about someone use the table between the -CREATE and the ALTER TABLE RELIABILITY. One solution would be to allow -the RELIABILITY as part of the CREATE TABLE, another is to tell users to -create the table inside a transaction. - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 359-1001 - + If your life is a hard drive, | 13 Roberts Road - + Christ can be your backup. | Newtown Square, Pennsylvania 19073 - ----------------------------(end of broadcast)--------------------------- -TIP 9: In versions below 8.0, the planner will ignore your desire to - choose an index scan if your joining column's datatypes do not - match - -From pgsql-hackers-owner+M78036=pgman=candle.pha.pa.us@postgresql.org Thu Dec 29 12:21:12 2005 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -To: Andrew Dunstan -cc: , , , - , , , - , -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -References: <200512290158.jBT1wEK28785@candle.pha.pa.us> - <51082.68.143.134.146.1135872877.squirrel@www.dunslane.net> -In-Reply-To: <51082.68.143.134.146.1135872877.squirrel@www.dunslane.net> -From: Greg Stark -Organization: The Emacs Conspiracy; member since 1992 -Date: 29 Dec 2005 12:20:32 -0500 -Message-ID: <87vex74y73.fsf@stark.xeocode.com> -Lines: 42 -User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.4 -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.112 required=5 tests=[AWL=0.112] -X-Spam-Score: 0.112 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 1983 - -"Andrew Dunstan" writes: - -> Bruce Momjian said: -> > DROP would drop the table on a restart -> > after a non-clean shutdown. It would do _no_ logging on the table and -> > allow concurrent access, plus index access. DELETE is the same as -> > DROP, but it just truncates the table (perhaps TRUNCATE is a better -> > word). -> > -> > EXCLUSIVE would allow only a single session to modify the table, and -> > would do all changes by appending to the table, similar to COPY LOCK. -> > EXCLUSIVE would also not allow indexes because those can not be -> > isolated like appending to the heap. EXCLUSIVE would write all dirty -> > shared buffers for the table and fsync them before committing. SHARE -> > is the functionality we have now, with full logging. -> -> I an horribly scared that this will be used as a "performance boost" for -> normal use. I would at least like to see some restrictions that make it -> harder to mis-use. Perhaps restrict to superuser? - -Well that's its whole purpose. At least you can hardly argue that you didn't -realize the consequences of "DELETE ROWS ON RECOVERY"... :) - -Some thoughts: - -a) I'm not sure I understand the purpose of EXCLUSIVE. When would I ever want to - use it instead of DELETE ROWS? - -b) It seems like the other feature people were talking about of not logging - for a table created within the same transaction should be handled by - having this flag implicitly set for any such newly created table. - Ie, the test for whether to log would look like: - - if (!table->logged && table->xid != myxid) ... - -c) Every option in ALTER TABLE should be in CREATE TABLE as well. - -d) Yes as someone else mentioned, this should only be allowable on a table - with no foreign keys referencing it. - --- -greg - - ----------------------------(end of broadcast)--------------------------- -TIP 9: In versions below 8.0, the planner will ignore your desire to - choose an index scan if your joining column's datatypes do not - match - -From pgsql-hackers-owner+M78037=pgman=candle.pha.pa.us@postgresql.org Thu Dec 29 12:31:40 2005 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -From: Bruce Momjian -Message-ID: <200512291730.jBTHUnn09840@candle.pha.pa.us> -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -In-Reply-To: <87vex74y73.fsf@stark.xeocode.com> -To: Greg Stark -Date: Thu, 29 Dec 2005 12:30:49 -0500 (EST) -cc: Andrew Dunstan , kleptog@svana.org, - simon@2ndquadrant.com, tgl@sss.pgh.pa.us, pg@rbt.ca, zhouqq@cs.toronto.edu, - pgsql-hackers@postgresql.org -X-Mailer: ELM [version 2.4ME+ PL121 (25)] -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.122 required=5 tests=[AWL=0.122] -X-Spam-Score: 0.122 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 3304 - -Greg Stark wrote: -> "Andrew Dunstan" writes: -> -> > Bruce Momjian said: -> > > DROP would drop the table on a restart -> > > after a non-clean shutdown. It would do _no_ logging on the table and -> > > allow concurrent access, plus index access. DELETE is the same as -> > > DROP, but it just truncates the table (perhaps TRUNCATE is a better -> > > word). -> > > -> > > EXCLUSIVE would allow only a single session to modify the table, and -> > > would do all changes by appending to the table, similar to COPY LOCK. -> > > EXCLUSIVE would also not allow indexes because those can not be -> > > isolated like appending to the heap. EXCLUSIVE would write all dirty -> > > shared buffers for the table and fsync them before committing. SHARE -> > > is the functionality we have now, with full logging. -> > -> > I an horribly scared that this will be used as a "performance boost" for -> > normal use. I would at least like to see some restrictions that make it -> > harder to mis-use. Perhaps restrict to superuser? -> -> Well that's its whole purpose. At least you can hardly argue that you didn't -> realize the consequences of "DELETE ROWS ON RECOVERY"... :) - -True. I think we are worried about non-owners using it, but the owner -had to grant permissions for others to modify it, so we might be OK. - -> Some thoughts: -> -> a) I'm not sure I understand the purpose of EXCLUSIVE. When would I ever want to -> use it instead of DELETE ROWS? - -Good question. The use case is doing COPY into a table that already had -data. EXCLUSIVE allows additions to the table but preserves the -existing data on a crash. - -> b) It seems like the other feature people were talking about of not logging -> for a table created within the same transaction should be handled by -> having this flag implicitly set for any such newly created table. -> Ie, the test for whether to log would look like: -> -> if (!table->logged && table->xid != myxid) ... - -Yes, the question is whether we want to limit users to having this -optimization _only_ when they have created the table in the same -transaction, and the short answer is we don't. - -> c) Every option in ALTER TABLE should be in CREATE TABLE as well. - -I looked into that and see that things like: - - ALTER [ COLUMN ] column SET STATISTICS integer - ALTER [ COLUMN ] column SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN } - -are not supported by CREATE TABLE, and probably shouldn't be because the -value can be changed after the table is created. I think the only -things we usually support in CREATE TABLE are those that cannot be -altered. - -> d) Yes as someone else mentioned, this should only be allowable on a table -> with no foreign keys referencing it. - -Right, and EXCLUSIVE can not have an index either. - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 359-1001 - + If your life is a hard drive, | 13 Roberts Road - + Christ can be your backup. | Newtown Square, Pennsylvania 19073 - ----------------------------(end of broadcast)--------------------------- -TIP 1: if posting/reading through Usenet, please send an appropriate - subscribe-nomail command to majordomo@postgresql.org so that your - message can get through to the mailing list cleanly - -From simon@2ndquadrant.com Fri Dec 30 08:10:53 2005 -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -From: Simon Riggs -To: Bruce Momjian -cc: Andrew Dunstan , Tom Lane , - Martijn van Oosterhout , Greg Stark , - Rod Taylor , Qingqing Zhou , - pgsql-hackers@postgresql.org -In-Reply-To: <200512291637.jBTGbdC03848@candle.pha.pa.us> -References: <200512291637.jBTGbdC03848@candle.pha.pa.us> -Date: Fri, 30 Dec 2005 13:09:12 +0000 -Message-ID: <1135948152.2862.113.camel@localhost.localdomain> -X-Mailer: Evolution 2.2.3 (2.2.3-2.fc4) -Content-Length: 6343 - -On Thu, 2005-12-29 at 11:37 -0500, Bruce Momjian wrote: -> Tom Lane wrote: -> > Simon Riggs writes: -> > > My view would be that this thread has been complex because everybody has -> > > expressed a somewhat different requirement, which could be broken down -> > > as: -> > > 1. The need for a multi-user-accessible yet temporary table -> > > 2. Loading data into a table immediately after it is created (i.e. in -> > > same transaction), including but not limited to a reload from pg_dump -> > > 3. How to load data quickly into an existing table (COPY) -> > > 4. How to add/modify data quickly in an existing table (INSERT SELECT, -> > > UPDATE) - -> > However, you then seem to be arguing for still using the COPY LOCK -> > syntax, which I think Bruce intended would go away in favor of using -> > these ALTER commands. Certainly that's what I'd prefer --- COPY has -> > got too darn many options already. - -COPY LOCK was Tom's suggestion at the end of a long discussion thread on -this precise issue. Nobody objected to it at that point; I implemented -it *exactly* that way because I wanted to very visibly follow the -consensus of the community, after informed debate. -http://archives.postgresql.org/pgsql-hackers/2005-06/msg00068.php - -Please re-read the links to previous discussions. -http://archives.postgresql.org/pgsql-hackers/2005-06/msg00069.php -There are points there, not made by me, that still apply and need to be -considered here, yet have not been. - -Just to restate my current thinking: -- agree we should have ALTER TABLE ... RELIABILITY DELETE ROWS -- we should have COPY LOCK rather than -ALTER TABLE .... RELIABILITY EXCLUSIVE -(Though I welcome better wording and syntax in either case; it is the -behaviour only that I discuss). - -It seems now that we have agreed approaches for (1), (2) and (4). Please -note that I have listened to the needs of others with regard to -requirement (1), as espoused by earlier by Hannu and again now by -Martijn. Some of the points about requirement (3) I made in my previous -post have not yet been addressed, IMHO. - -My mind is not fixed. AFAICS there are valid points remaining on both -sides of the discussion about loading data quickly into an existing -table. - -> I do think it is valid concern about someone use the table between the -> CREATE and the ALTER TABLE RELIABILITY. One solution would be to allow -> the RELIABILITY as part of the CREATE TABLE, another is to tell users to -> create the table inside a transaction. - -Neither solution works for this use case: - -> > 3. How to load data quickly into an existing table (COPY) - -This is the only use case for which ALTER TABLE ... EXCLUSIVE makes -sense. That option means that any write lock held upon the table would -be an EXCLUSIVE table lock, so would never be a performance gain with -single row INSERT, UPDATE or DELETEs. - -Following Andrew's concerns, I'd also note that ALTER TABLE requires a -much higher level of privilege to operate than does COPY. That sounds -like it will make things more secure, but all it does is open up the -administrative rights, since full ownership rights must be obtained -merely to load data. - -> Having COPY behave differently because it is -> in a transaction is fine as long as it is user-invisible - -Good - -> I think there is great utility in giving users one API, namely -> RELIABILITY (or some other keyword), and telling them that is where they -> control logging. I realize adding one keyword, LOCK, to an existing -> command isn't a big deal, but once you decentralize your API enough -> times, you end up with a terribly complex database system. It is this -> design rigidity that helps make PostgreSQL so much easier to use than -> other database systems. - -I do see the appeal of your suggestion... - -TRUNCATE is a special command to delete quickly. There is no requirement -to do an ALTER TABLE statement before that command executes. - -Balance would suggest that a special command to load data quickly would -be reasonably accepted by users. - - - - -Minor points below: - -> > > In the patch, pg_dump has *not* been altered to use COPY LOCK, so a -> > > pg_dump *will* work with any other version of PostgreSQL, which *would -> > > not* be the case if we added ALTER TABLE ... RELIABILITY statements into -> > > it. -> > -> > Wrong --- the good thing about ALTER TABLE is that an old version of -> > Postgres would simply reject it and keep going. Therefore we could get -> > the speedup in dumps without losing compatibility, which is not true -> > of COPY LOCK. - -That was pointing out one of Bruce's objections was not relevant because -it assumed COPY LOCK was required to make pg_restore go faster; that was -not the case - so there is no valid objection either way now. - -> > BTW, this is a perfect example of the use-case for not abandoning a -> > dump-file load simply because one command fails. (We have relied on -> > this sort of reasoning many times before, too, for example by using -> > "SET default_with_oids" in preference to CREATE TABLE WITH/WITHOUT OIDS.) -> > I don't think that "wrap the whole load into begin/end" is really a very -> > workable answer, because there are far too many scenarios where you -> > can't do that. Another one where it doesn't help is a data-only dump. - -Which is why --single-transaction is not the default, per the earlier -discussion on that point (on -patches). - -> Yep, Tom is echoing my reaction. There is a temptation to add things up -> onto existing commands, e.g. LOCK, and while it works, it makes for some -> very complex user API's. Having COPY behave differently because it is -> in a transaction is fine as long as it is user-invisible, but once you -> require users to do that to get the speedup, it isn't user-invisible -> anymore. -> -> (I can see it now, "Why is pg_dump putting things in transactions?", -> "Because it prevents it from being logged." "Oh, should I be doing that -> in my code?" "Perhaps, if you want ..." You can see where that -> discussion is going. Having them see "ATER TABLE ... RELIBILITY -> TRUNCATE" is very clear, and very clear on how it can be used in user -> code.) - -The above case is not an argument against COPY LOCK. Exactly what you -say above would still occur even when we have ALTER TABLE ... -RELIABILITY statement, since COPY LOCK and -COPY-optimized-within-same-transaction are different things. - -Best Regards, Simon Riggs - -From pgsql-hackers-owner+M78064=pgman=candle.pha.pa.us@postgresql.org Fri Dec 30 11:50:49 2005 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -From: Bruce Momjian -Message-ID: <200512301649.jBUGnxn21488@candle.pha.pa.us> -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -In-Reply-To: <1135948152.2862.113.camel@localhost.localdomain> -To: Simon Riggs -Date: Fri, 30 Dec 2005 11:49:59 -0500 (EST) -cc: Andrew Dunstan , Tom Lane , - Martijn van Oosterhout , Greg Stark , - Rod Taylor , Qingqing Zhou , - pgsql-hackers@postgresql.org -X-Mailer: ELM [version 2.4ME+ PL121 (25)] -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.12 required=5 tests=[AWL=0.120] -X-Spam-Score: 0.12 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 8888 - -Simon Riggs wrote: -> On Thu, 2005-12-29 at 11:37 -0500, Bruce Momjian wrote: -> > Tom Lane wrote: -> > > Simon Riggs writes: -> > > > My view would be that this thread has been complex because everybody has -> > > > expressed a somewhat different requirement, which could be broken down -> > > > as: -> > > > 1. The need for a multi-user-accessible yet temporary table -> > > > 2. Loading data into a table immediately after it is created (i.e. in -> > > > same transaction), including but not limited to a reload from pg_dump -> > > > 3. How to load data quickly into an existing table (COPY) -> > > > 4. How to add/modify data quickly in an existing table (INSERT SELECT, -> > > > UPDATE) -> -> > > However, you then seem to be arguing for still using the COPY LOCK -> > > syntax, which I think Bruce intended would go away in favor of using -> > > these ALTER commands. Certainly that's what I'd prefer --- COPY has -> > > got too darn many options already. -> -> COPY LOCK was Tom's suggestion at the end of a long discussion thread on -> this precise issue. Nobody objected to it at that point; I implemented -> it *exactly* that way because I wanted to very visibly follow the -> consensus of the community, after informed debate. -> http://archives.postgresql.org/pgsql-hackers/2005-06/msg00068.php -> -> Please re-read the links to previous discussions. -> http://archives.postgresql.org/pgsql-hackers/2005-06/msg00069.php -> There are points there, not made by me, that still apply and need to be -> considered here, yet have not been. - -Yes, I know we agreed to the COPY LOCK, but new features now being -requested, so we have to re-evaluate where we are going with COPY LOCK -to get a more consistent solution. - -> Just to restate my current thinking: -> - agree we should have ALTER TABLE ... RELIABILITY DELETE ROWS -> - we should have COPY LOCK rather than -> ALTER TABLE .... RELIABILITY EXCLUSIVE -> (Though I welcome better wording and syntax in either case; it is the -> behaviour only that I discuss). -> -> It seems now that we have agreed approaches for (1), (2) and (4). Please -> note that I have listened to the needs of others with regard to -> requirement (1), as espoused by earlier by Hannu and again now by -> Martijn. Some of the points about requirement (3) I made in my previous -> post have not yet been addressed, IMHO. -> -> My mind is not fixed. AFAICS there are valid points remaining on both -> sides of the discussion about loading data quickly into an existing -> table. -> -> > I do think it is valid concern about someone use the table between the -> > CREATE and the ALTER TABLE RELIABILITY. One solution would be to allow -> > the RELIABILITY as part of the CREATE TABLE, another is to tell users to -> > create the table inside a transaction. -> -> Neither solution works for this use case: -> -> > > 3. How to load data quickly into an existing table (COPY) -> -> This is the only use case for which ALTER TABLE ... EXCLUSIVE makes -> sense. That option means that any write lock held upon the table would -> be an EXCLUSIVE table lock, so would never be a performance gain with -> single row INSERT, UPDATE or DELETEs. - -Ah, but people wanted fast INSERT INTO ... SELECT, and that would use -EXCLUSIVE too. What about a massive UPDATE? Perhaps that could use -EXCLUSIVE? We don't want to add "LOCK" to every command that might use -EXCLUSIVE. ALTER is much better for this. - -I agree if we thought EXCLUSIVE would only be used for COPY, we could -use LOCK, but I am thinking it will be used for other commands as well. - -> Following Andrew's concerns, I'd also note that ALTER TABLE requires a -> much higher level of privilege to operate than does COPY. That sounds -> like it will make things more secure, but all it does is open up the -> administrative rights, since full ownership rights must be obtained -> merely to load data. - -True, but as pointed out by others, I don't see that happening too -often. - -> > Having COPY behave differently because it is -> > in a transaction is fine as long as it is user-invisible -> -> Good -> -> > I think there is great utility in giving users one API, namely -> > RELIABILITY (or some other keyword), and telling them that is where they -> > control logging. I realize adding one keyword, LOCK, to an existing -> > command isn't a big deal, but once you decentralize your API enough -> > times, you end up with a terribly complex database system. It is this -> > design rigidity that helps make PostgreSQL so much easier to use than -> > other database systems. -> -> I do see the appeal of your suggestion... -> -> TRUNCATE is a special command to delete quickly. There is no requirement -> to do an ALTER TABLE statement before that command executes. - -The TRUNCATE happens during recovery. There is no user interaction. It -happens because we can't restore the contents of the table in a -consistent state because no logging was used. Basically, a table marked -RELIABILITY TRUNCATE would be truncated on a recovery start of the -postmaster. - -> Balance would suggest that a special command to load data quickly would -> be reasonably accepted by users. -> -> -> -> -> Minor points below: -> -> > > > In the patch, pg_dump has *not* been altered to use COPY LOCK, so a -> > > > pg_dump *will* work with any other version of PostgreSQL, which *would -> > > > not* be the case if we added ALTER TABLE ... RELIABILITY statements into -> > > > it. -> > > -> > > Wrong --- the good thing about ALTER TABLE is that an old version of -> > > Postgres would simply reject it and keep going. Therefore we could get -> > > the speedup in dumps without losing compatibility, which is not true -> > > of COPY LOCK. -> -> That was pointing out one of Bruce's objections was not relevant because -> it assumed COPY LOCK was required to make pg_restore go faster; that was -> not the case - so there is no valid objection either way now. - -I don't consider the single-transaction to be a no-cost solution. You -are adding flags to commands, and you are using a dump layout for -performance where the purpose for the layout is not clear. The ALTER is -clear to the user, and it allows nologging operations to happen after -the table is created. - -In fact, for use in pg_dump, I think DROP is the proper operation for -loading, not your transaction wrapping solution. We already agree we -need DROP (or TRUNCATE), so why not use that rather than the transaction -wrap idea? - -> > > BTW, this is a perfect example of the use-case for not abandoning a -> > > dump-file load simply because one command fails. (We have relied on -> > > this sort of reasoning many times before, too, for example by using -> > > "SET default_with_oids" in preference to CREATE TABLE WITH/WITHOUT OIDS.) -> > > I don't think that "wrap the whole load into begin/end" is really a very -> > > workable answer, because there are far too many scenarios where you -> > > can't do that. Another one where it doesn't help is a data-only dump. -> -> Which is why --single-transaction is not the default, per the earlier -> discussion on that point (on -patches). - -Right, but why not use DROP/TRUNCATE? That works for old dumps too, and -has no downsides, meaning it can be always on. - -> > Yep, Tom is echoing my reaction. There is a temptation to add things up -> > onto existing commands, e.g. LOCK, and while it works, it makes for some -> > very complex user API's. Having COPY behave differently because it is -> > in a transaction is fine as long as it is user-invisible, but once you -> > require users to do that to get the speedup, it isn't user-invisible -> > anymore. -> > -> > (I can see it now, "Why is pg_dump putting things in transactions?", -> > "Because it prevents it from being logged." "Oh, should I be doing that -> > in my code?" "Perhaps, if you want ..." You can see where that -> > discussion is going. Having them see "ATER TABLE ... RELIBILITY -> > TRUNCATE" is very clear, and very clear on how it can be used in user -> > code.) -> -> The above case is not an argument against COPY LOCK. Exactly what you -> say above would still occur even when we have ALTER TABLE ... -> RELIABILITY statement, since COPY LOCK and -> COPY-optimized-within-same-transaction are different things. - -See my posting above that we might want EXCLUSIVE for other commands, -meaning ALTER makes more sense. - -So, to summarize, I think we should add DROP/TRUNCATE, and use that by -default (or optionally off?) in pg_dump, and, assuming we want EXCLUSIVE -for more than just COPY, we need to add ALTER TABLE EXCLUSIVE. - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 359-1001 - + If your life is a hard drive, | 13 Roberts Road - + Christ can be your backup. | Newtown Square, Pennsylvania 19073 - ----------------------------(end of broadcast)--------------------------- -TIP 5: don't forget to increase your free space map settings - -From pgsql-hackers-owner+M78065=pgman=candle.pha.pa.us@postgresql.org Fri Dec 30 12:40:48 2005 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -Message-ID: <43B570C9.6060406@dunslane.net> -Date: Fri, 30 Dec 2005 12:39:21 -0500 -From: Andrew Dunstan -User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20050922 Fedora/1.7.12-1.3.1 -X-Accept-Language: en-us, en -To: Tom Lane -cc: simon@2ndquadrant.com, pgman@candle.pha.pa.us, kleptog@svana.org, - gsstark@mit.edu, pg@rbt.ca, zhouqq@cs.toronto.edu, - pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -References: <1135948152.2862.113.camel@localhost.localdomain> <56737.68.143.134.146.1135954413.squirrel@www.dunslane.net> <11876.1135954626@sss.pgh.pa.us> -In-Reply-To: <11876.1135954626@sss.pgh.pa.us> -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.041 required=5 tests=[AWL=0.041] -X-Spam-Score: 0.041 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 1815 - - - -Tom Lane wrote: - ->"Andrew Dunstan" writes: -> -> ->>Simon Riggs said: ->> ->> ->>>Following Andrew's concerns, I'd also note that ALTER TABLE requires a ->>>much higher level of privilege to operate than does COPY. That sounds ->>>like it will make things more secure, but all it does is open up the ->>>administrative rights, since full ownership rights must be obtained ->>>merely to load data. ->>> ->>> -> -> -> ->>My concern is more about making plain that this is for special operations, ->>not normal operations. Or maybe I have misunderstood the purpose. ->> ->> -> ->Rephrase that as "full ownership rights must be obtained to load data in ->a way that requires dropping any existing indexes and locking out other ->users of the table". I don't think the use-case for this will be very ->large for non-owners, or indeed even for owners except during initial ->table creation; and so I don't think the above argument is strong. -> -> -> -> - -Those restrictions aren't true of Bruce's proposed drop and -delete/truncate recovery modes, are they? - -People do crazy things in pursuit of performance. Illustration: a few -months ago I was instrumenting an app (based on MySQL/ISAM) and I -noticed that under load it simply didn't update the inventory properly - -of 1000 orders placed within a few seconds it might reduce inventory by -3 or 4. I reported this and they shrugged their shoulders and said -"well, we'd have to lock the table and that would slow everything down -...". - -I just want to be sure we aren't providing a footgun. "Oh, just set -recovery mode to delete. It won't make any difference unless you crash -and you'll run faster." - -cheers - -andrew - - - - ----------------------------(end of broadcast)--------------------------- -TIP 5: don't forget to increase your free space map settings - -From pgsql-hackers-owner+M78066=pgman=candle.pha.pa.us@postgresql.org Fri Dec 30 12:58:52 2005 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -From: Bruce Momjian -Message-ID: <200512301758.jBUHwFv03107@candle.pha.pa.us> -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -In-Reply-To: <43B570C9.6060406@dunslane.net> -To: Andrew Dunstan -Date: Fri, 30 Dec 2005 12:58:15 -0500 (EST) -cc: Tom Lane , simon@2ndquadrant.com, kleptog@svana.org, - gsstark@mit.edu, pg@rbt.ca, zhouqq@cs.toronto.edu, - pgsql-hackers@postgresql.org -X-Mailer: ELM [version 2.4ME+ PL121 (25)] -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.12 required=5 tests=[AWL=0.120] -X-Spam-Score: 0.12 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 1996 - -Andrew Dunstan wrote: -> >>My concern is more about making plain that this is for special operations, -> >>not normal operations. Or maybe I have misunderstood the purpose. -> >> -> >> -> > -> >Rephrase that as "full ownership rights must be obtained to load data in -> >a way that requires dropping any existing indexes and locking out other -> >users of the table". I don't think the use-case for this will be very -> >large for non-owners, or indeed even for owners except during initial -> >table creation; and so I don't think the above argument is strong. -> > -> > -> > -> > -> -> Those restrictions aren't true of Bruce's proposed drop and -> delete/truncate recovery modes, are they? - -Only the owner could do the ALTER, for sure, but once the owner sets it, -any user with permission to write to the table would have those -characteristics. - -> People do crazy things in pursuit of performance. Illustration: a few -> months ago I was instrumenting an app (based on MySQL/ISAM) and I -> noticed that under load it simply didn't update the inventory properly - -> of 1000 orders placed within a few seconds it might reduce inventory by -> 3 or 4. I reported this and they shrugged their shoulders and said -> "well, we'd have to lock the table and that would slow everything down -> ...". -> -> I just want to be sure we aren't providing a footgun. "Oh, just set -> recovery mode to delete. It won't make any difference unless you crash -> and you'll run faster." - -I think we have to trust the object owner in this case. I don't know of -any super-user-only ALTER commands, but I suppose we could set it up -that way if we wanted. - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 359-1001 - + If your life is a hard drive, | 13 Roberts Road - + Christ can be your backup. | Newtown Square, Pennsylvania 19073 - ----------------------------(end of broadcast)--------------------------- -TIP 2: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M78070=pgman=candle.pha.pa.us@postgresql.org Fri Dec 30 14:29:06 2005 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -From: Simon Riggs -To: Bruce Momjian -cc: Andrew Dunstan , Tom Lane , - Martijn van Oosterhout , Greg Stark , - Rod Taylor , Qingqing Zhou , - pgsql-hackers@postgresql.org -In-Reply-To: <200512301649.jBUGnxn21488@candle.pha.pa.us> -References: <200512301649.jBUGnxn21488@candle.pha.pa.us> -Date: Fri, 30 Dec 2005 19:28:41 +0000 -Message-ID: <1135970921.5052.68.camel@localhost.localdomain> -X-Mailer: Evolution 2.2.3 (2.2.3-2.fc4) -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.034 required=5 tests=[AWL=0.034] -X-Spam-Score: 0.034 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 3112 - -On Fri, 2005-12-30 at 11:49 -0500, Bruce Momjian wrote: - -> Yes, I know we agreed to the COPY LOCK, but new features now being -> requested, so we have to re-evaluate where we are going with COPY LOCK -> to get a more consistent solution. - -Thank you. - -> Ah, but people wanted fast INSERT INTO ... SELECT, and that would use -> EXCLUSIVE too. What about a massive UPDATE? Perhaps that could use -> EXCLUSIVE? We don't want to add "LOCK" to every command that might use -> EXCLUSIVE. ALTER is much better for this. - -> I agree if we thought EXCLUSIVE would only be used for COPY, we could -> use LOCK, but I am thinking it will be used for other commands as well. - -Agreed, I will look to implement this. - -Could the internals of my recent patch be reviewed? Changing the user -interface is less of a problem than changing the internals, which is -where the hard work takes place. I do not want to extend this work -further only to have that part rejected later. - -The implications of EXCLUSIVE are: -- there will be a check on each and every I, U, D to check the state of -the relation -- *every* operation that attempts a write lock will attempt to acquire -an EXCLUSIVE full table lock instead -- following successful completion of *each* DML statement, the relation -will be heap_sync'd involving a full scan of the buffer cache - -Can I clarify the wording of the syntax? Is EXCLUSIVE the right word? -How about FASTLOAD or BULKLOAD? Those words seem less likely to be -misused in the future - i.e. we are invoking a special mode, rather than -invoking a special "go faster" option. - -> I don't consider the single-transaction to be a no-cost solution. You -> are adding flags to commands, and you are using a dump layout for -> performance where the purpose for the layout is not clear. The ALTER is -> clear to the user, and it allows nologging operations to happen after -> the table is created. -> -> In fact, for use in pg_dump, I think DROP is the proper operation for -> loading, not your transaction wrapping solution. We already agree we -> need DROP (or TRUNCATE), so why not use that rather than the transaction -> wrap idea? - -This was discussed on-list by 2 core team members, a committer and -myself, but I see no requirements change here. You even accepted the -invisible COPY optimization in your last post - why unpick that now? -Please forgive my tone, but I am lost for reasonable yet expressive -words. - -The --single-transaction mode would apply even if the dump was created -using an earlier version of pg_dump. pg_dump has *not* been altered at -all. (And I would again add that the idea was not my own) - -> So, to summarize, I think we should add DROP/TRUNCATE, and use that by -> default (or optionally off?) in pg_dump, and, assuming we want EXCLUSIVE -> for more than just COPY, we need to add ALTER TABLE EXCLUSIVE. - -Would you mind stating again what you mean, just so I can understand -this? Your summary isn't enough. - -Best Regards, Simon Riggs - - ----------------------------(end of broadcast)--------------------------- -TIP 3: Have you checked our extensive FAQ? - - http://www.postgresql.org/docs/faq - -From pgsql-hackers-owner+M78072=pgman=candle.pha.pa.us@postgresql.org Fri Dec 30 16:15:30 2005 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -From: Bruce Momjian -Message-ID: <200512302114.jBULEno02301@candle.pha.pa.us> -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -In-Reply-To: <1135970921.5052.68.camel@localhost.localdomain> -To: Simon Riggs -Date: Fri, 30 Dec 2005 16:14:49 -0500 (EST) -cc: Andrew Dunstan , Tom Lane , - Martijn van Oosterhout , Greg Stark , - Rod Taylor , Qingqing Zhou , - pgsql-hackers@postgresql.org -X-Mailer: ELM [version 2.4ME+ PL121 (25)] -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.12 required=5 tests=[AWL=0.120] -X-Spam-Score: 0.12 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 7285 - -Simon Riggs wrote: -> On Fri, 2005-12-30 at 11:49 -0500, Bruce Momjian wrote: -> -> > Yes, I know we agreed to the COPY LOCK, but new features now being -> > requested, so we have to re-evaluate where we are going with COPY LOCK -> > to get a more consistent solution. -> -> Thank you. - -Good. I think we can be happy that COPY LOCK didn't get into a release, -so we don't have to support it forever. When we are adding features, we -have to consider not only the current release, but future releases and -what people will ask for in the future so the syntax can be expanded -without breaking previous usage. - -> > Ah, but people wanted fast INSERT INTO ... SELECT, and that would use -> > EXCLUSIVE too. What about a massive UPDATE? Perhaps that could use -> > EXCLUSIVE? We don't want to add "LOCK" to every command that might use -> > EXCLUSIVE. ALTER is much better for this. -> -> > I agree if we thought EXCLUSIVE would only be used for COPY, we could -> > use LOCK, but I am thinking it will be used for other commands as well. -> -> Agreed, I will look to implement this. -> -> Could the internals of my recent patch be reviewed? Changing the user -> interface is less of a problem than changing the internals, which is -> where the hard work takes place. I do not want to extend this work -> further only to have that part rejected later. - -OK, I will look it over this week or next. - -> The implications of EXCLUSIVE are: -> - there will be a check on each and every I, U, D to check the state of -> the relation -> - *every* operation that attempts a write lock will attempt to acquire -> an EXCLUSIVE full table lock instead -> - following successful completion of *each* DML statement, the relation -> will be heap_sync'd involving a full scan of the buffer cache - -Yes, I think that is it. What we can do is implement EXCLUSIVE to -affect only COPY at this point, and document that, and later add other -commands. - -> Can I clarify the wording of the syntax? Is EXCLUSIVE the right word? -> How about FASTLOAD or BULKLOAD? Those words seem less likely to be -> misused in the future - i.e. we are invoking a special mode, rather than -> invoking a special "go faster" option. - -The problem with the FASTLOAD/BULKLOAD words is that EXCLUSIVE mode is -probably not the best for loading. I would think TRUNCATE would be a -better option. - -In fact, in loading a table, I think both EXCLUSIVE and TRUNCATE would be -the same, mostly. You would create the table, set its RELIABILITY to -TRUNCATE, COPY into the table, then set the RELIABILITY to SHARE or -DEFAULT. The second ALTER has to sync all the dirty data blocks, which -the same thing EXCLUSIVE does at the conclusion of COPY. - -So, we need a name for EXCLUSIVE mode that suggests how it is different -from TRUNCATE, and in this case, the difference is that EXCLUSIVE -preserves the previous contents of the table on recovery, while TRUNCATE -does not. Do you want to call the mode PRESERVE, or EXCLUSIVE WRITER? -Anyway, the keywords are easy to modify, even after the patch is -submitted. FYI, I usually go through keywords.c looking for a keyword -we already use. - -> > I don't consider the single-transaction to be a no-cost solution. You -> > are adding flags to commands, and you are using a dump layout for -> > performance where the purpose for the layout is not clear. The ALTER is -> > clear to the user, and it allows nologging operations to happen after -> > the table is created. -> > -> > In fact, for use in pg_dump, I think DROP is the proper operation for -> > loading, not your transaction wrapping solution. We already agree we -> > need DROP (or TRUNCATE), so why not use that rather than the transaction -> > wrap idea? -> -> This was discussed on-list by 2 core team members, a committer and -> myself, but I see no requirements change here. You even accepted the -> invisible COPY optimization in your last post - why unpick that now? -> Please forgive my tone, but I am lost for reasonable yet expressive -> words. - -Do you think you are the only one who has rewritten a patch multiple -times? We all have. The goal is to get the functionality into the -system in the most seamless way possible. Considering the number of -people who use PostgreSQL, if it takes use 10 tries, it is worth it -considering the thousands of people who will use it. Would you have us -include a sub-optimal patch and have thousands of people adjust to its -non-optimal functionality? I am sure you would not. Perhaps a company -would say, "Oh, just ship it", but we don't. - -> The --single-transaction mode would apply even if the dump was created -> using an earlier version of pg_dump. pg_dump has *not* been altered at -> all. (And I would again add that the idea was not my own) - -I assume you mean this: - - http://archives.postgresql.org/pgsql-patches/2005-12/msg00257.php - -I guess with the ALTER commands I don't see much value in the ---single-transaction flag. I am sure others suggested it, but would -they suggest it now given our current direction. The fact that the -patch was submitted does not give it any more weight --- the question is -does this feature make sense for 8.2. The goal is not to cram as many -optimizations into PostgreSQL as possible, the goal is to present a -consistent usable system to users. - -> > So, to summarize, I think we should add DROP/TRUNCATE, and use that by -> > default (or optionally off?) in pg_dump, and, assuming we want EXCLUSIVE -> > for more than just COPY, we need to add ALTER TABLE EXCLUSIVE. -> -> Would you mind stating again what you mean, just so I can understand -> this? Your summary isn't enough. - -New ALTER TABLE mode, perhaps call it PERSISTENCE: - - ALTER TABLE tab PERSISTENCE DROP ON RECOVERY - ALTER TABLE tab PERSISTENCE TRUNCATE ON RECOVERY - -These would drop or truncate all tables with this flag on a non-clean -start of the postmaster, and write something in the server logs. -However, I don't know that we have the code in place to DROP/TRUNCATE in -recovery mode, and it would affect all databases, so it could be quite -complex to implement. In this mode, no WAL logs would be written for -table modifications, though DDL commands would have to be logged. - - ALTER TABLE tab PERSISTENCE PRESERVE (or STABLE?) - -Table contents are preserved across recoveries, but data modifications -can happen only one at a time. I don't think we have a lock mode that -does this, so I am worried a new lock mode will have to be created. A -simplified solution at this stage would be to take an exclusive lock on -the table, but really we just need a single-writer table lock, which I -don't think we have. initially this can implemented to only affect COPY -but later can be done for other commands. - - ALTER TABLE tab PERSISTENCE DEFAULT - -This would be our current default mode, which is full concurrency and -persistence. - -It took me over an hour to write this, but I feel the time is worth it -because of the number of users who use our software. - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 359-1001 - + If your life is a hard drive, | 13 Roberts Road - + Christ can be your backup. | Newtown Square, Pennsylvania 19073 - ----------------------------(end of broadcast)--------------------------- -TIP 2: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M78076=pgman=candle.pha.pa.us@postgresql.org Fri Dec 30 17:37:00 2005 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -To: Bruce Momjian -cc: Simon Riggs , Andrew Dunstan , - Tom Lane , Martijn van Oosterhout , - Greg Stark , Rod Taylor , - Qingqing Zhou , pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -References: <200512302114.jBULEno02301@candle.pha.pa.us> -In-Reply-To: <200512302114.jBULEno02301@candle.pha.pa.us> -From: Greg Stark -Organization: The Emacs Conspiracy; member since 1992 -Date: 30 Dec 2005 17:36:24 -0500 -Message-ID: <87mzii8b6f.fsf@stark.xeocode.com> -Lines: 28 -User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.4 -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.113 required=5 tests=[AWL=0.113] -X-Spam-Score: 0.113 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 1424 - - -As far as EXCLUSIVE or COPY LOCK goes, I think this would be useful -functionality but perhaps there doesn't have to be any proprietary user -interface to it at all. Why not just check if the conditions are already -present to allow the optimization and if so go ahead. - -That is, if the current transaction already has an exclusive lock on the table -and there are no indexes (and PITR isn't active) then Postgres could go ahead -and use the same WAL skipping logic as the other operations that already so -so. This would work for inserts whether coming from COPY or plain SQL INSERTs. - -The nice thing about this is that the user's SQL wouldn't need any proprietary -extensions at all. Just tell people to do - -BEGIN; -LOCK TABLE foo; -COPY foo from ... -COMMIT; - -There could be a COPY LOCK option to obtain a lock, but it would be purely for -user convenience so they don't have to bother with BEGIN and COMMIt. - -The only downside is a check to see if an exclusive table lock is present on -every copy and insert. That might be significant but perhaps there are ways to -finess that. If not perhaps only doing it on COPY would be a good compromise. - --- -greg - - ----------------------------(end of broadcast)--------------------------- -TIP 1: if posting/reading through Usenet, please send an appropriate - subscribe-nomail command to majordomo@postgresql.org so that your - message can get through to the mailing list cleanly - -From pgsql-hackers-owner+M78077=pgman=candle.pha.pa.us@postgresql.org Fri Dec 30 17:47:18 2005 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -From: Bruce Momjian -Message-ID: <200512302246.jBUMkjF25196@candle.pha.pa.us> -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -In-Reply-To: <87mzii8b6f.fsf@stark.xeocode.com> -To: Greg Stark -Date: Fri, 30 Dec 2005 17:46:45 -0500 (EST) -cc: Simon Riggs , Andrew Dunstan , - Tom Lane , Martijn van Oosterhout , - Rod Taylor , Qingqing Zhou , - pgsql-hackers@postgresql.org -X-Mailer: ELM [version 2.4ME+ PL121 (25)] -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.12 required=5 tests=[AWL=0.120] -X-Spam-Score: 0.12 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 2135 - -Greg Stark wrote: -> -> As far as EXCLUSIVE or COPY LOCK goes, I think this would be useful -> functionality but perhaps there doesn't have to be any proprietary user -> interface to it at all. Why not just check if the conditions are already -> present to allow the optimization and if so go ahead. -> -> That is, if the current transaction already has an exclusive lock on the table -> and there are no indexes (and PITR isn't active) then Postgres could go ahead -> and use the same WAL skipping logic as the other operations that already so -> so. This would work for inserts whether coming from COPY or plain SQL INSERTs. -> -> The nice thing about this is that the user's SQL wouldn't need any proprietary -> extensions at all. Just tell people to do -> -> BEGIN; -> LOCK TABLE foo; -> COPY foo from ... -> COMMIT; -> -> There could be a COPY LOCK option to obtain a lock, but it would be purely for -> user convenience so they don't have to bother with BEGIN and COMMIt. -> -> The only downside is a check to see if an exclusive table lock is present on -> every copy and insert. That might be significant but perhaps there are ways to -> finess that. If not perhaps only doing it on COPY would be a good compromise. - -Well, again, if we wanted to use EXCLUSIVE only for COPY, this might -make sense. However, also consider that the idea for EXCLUSIVE was that -users could continue read-only queries on the table while it is being -loaded (like COPY allows now), and that in EXCLUSIVE mode, we are only -going to write into new pages. - -If someone has an exclusive lock on the table and does a COPY or SELECT -INTO do we want to assume we are only going to write into new pages, and -do we want to force an exclusive lock rather than a single-writer lock? -I don't think so. - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 359-1001 - + If your life is a hard drive, | 13 Roberts Road - + Christ can be your backup. | Newtown Square, Pennsylvania 19073 - ----------------------------(end of broadcast)--------------------------- -TIP 6: explain analyze is your friend - -From mpaesold@gmx.at Sat Dec 31 06:59:51 2005 -Date: Sat, 31 Dec 2005 12:59:44 +0100 (MET) -From: Michael Paesold -To: Bruce Momjian -cc: simon@2ndquadrant.com, andrew@dunslane.net, tgl@sss.pgh.pa.us, - kleptog@svana.org, gsstark@mit.edu, pg@rbt.ca, zhouqq@cs.toronto.edu, - pgsql-hackers@postgresql.org -References: <200512302114.jBULEno02301@candle.pha.pa.us> -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -X-Priority: 3 (Normal) -X-Authenticated: #1946847 -Message-ID: <14969.1136030384@www6.gmx.net> -X-Mailer: WWW-Mail 1.6 (Global Message Exchange) -X-Flags: 0001 -Content-Length: 1305 - -Bruce Momjian wrote: - -> > The --single-transaction mode would apply even if the dump was created -> > using an earlier version of pg_dump. pg_dump has *not* been altered at -> > all. (And I would again add that the idea was not my own) -> -> I assume you mean this: -> -> http://archives.postgresql.org/pgsql-patches/2005-12/msg00257.php -> -> I guess with the ALTER commands I don't see much value in the -> --single-transaction flag. I am sure others suggested it, but would -> they suggest it now given our current direction. - -I just want to add that --single-transaction has a value of it's own. There -were times when I wanted to restore parts of a dump all-or-nothing. - -This is possible with PostgreSQL, unlike many other DBM systems, because -people like Tom Lane have invested in ensuring that all DDL is working -without implicitly committing an enclosing transaction. - -Using pg_restore directly into a database, it is not possible to get a -single transaction right now. One has to restore to a file and manually -added BEGIN/COMMIT. Just for that I think --single-transaction is a great -addition and a missing feature. - -I think more people have a use-case for that. - -Best Regards, -Michael Paesold - --- -Telefonieren Sie schon oder sparen Sie noch? -NEU: GMX Phone_Flat http://www.gmx.net/de/go/telefonie - -From pgsql-hackers-owner+M78213=pgman=candle.pha.pa.us@postgresql.org Tue Jan 3 12:08:43 2006 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -From: Bruce Momjian -Message-ID: <200601031708.k03H85j27170@candle.pha.pa.us> -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -In-Reply-To: <17173.1136306881@sss.pgh.pa.us> -To: Tom Lane -Date: Tue, 3 Jan 2006 12:08:05 -0500 (EST) -cc: Jim C. Nasby , - Andrew Dunstan , simon@2ndquadrant.com, - kleptog@svana.org, gsstark@mit.edu, pg@rbt.ca, zhouqq@cs.toronto.edu, - pgsql-hackers@postgresql.org -X-Mailer: ELM [version 2.4ME+ PL121 (25)] -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.121 required=5 tests=[AWL=0.121] -X-Spam-Score: 0.121 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 1125 - -Tom Lane wrote: -> "Jim C. Nasby" writes: -> > On Tue, Jan 03, 2006 at 11:26:51AM -0500, Tom Lane wrote: -> >> Such an ALTER would certainly require exclusive lock on the table, -> >> so I'm not sure that I see much use-case for doing it like that. -> >> You'd want to do the ALTER and commit so as not to lock other people -> >> out of the table entirely while doing the bulk data-pushing. -> -> > Maybe this just isn't clear, but would EXCLUSIVE block writes from all -> > other sessions then? -> -> I don't think it should (which implies that EXCLUSIVE is a bad name). - -Agreed, EXCLUSIVE was used to mean an _exclusive_ writer. The new words -I proposed were PRESERVE or STABLE. - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 359-1001 - + If your life is a hard drive, | 13 Roberts Road - + Christ can be your backup. | Newtown Square, Pennsylvania 19073 - ----------------------------(end of broadcast)--------------------------- -TIP 4: Have you searched our list archives? - - http://archives.postgresql.org - -From tgl@sss.pgh.pa.us Tue Jan 3 12:37:34 2006 -To: Stephen Frost -cc: Jim C. Nasby , - Bruce Momjian , - Andrew Dunstan , kleptog@svana.org, - simon@2ndquadrant.com, gsstark@mit.edu, pg@rbt.ca, zhouqq@cs.toronto.edu, - pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -In-Reply-To: <20060103165359.GP6026@ns.snowman.net> -References: <200512291605.jBTG5gi00396@candle.pha.pa.us> <7966.1135873468@sss.pgh.pa.us> <20060103154521.GC82560@pervasive.com> <20060103162137.GO6026@ns.snowman.net> <16856.1136305742@sss.pgh.pa.us> <20060103165359.GP6026@ns.snowman.net> -Comments: In-reply-to Stephen Frost - message dated "Tue, 03 Jan 2006 11:54:01 -0500" -Date: Tue, 03 Jan 2006 12:37:32 -0500 -Message-ID: <17841.1136309852@sss.pgh.pa.us> -From: Tom Lane -Content-Length: 976 - -Stephen Frost writes: -> The problem is that you might want to grant 'truncate' to people who -> *aren't* particularly trusted. For truncate, at least I have a -> real-world use-case for it. - -I don't find this use-case particularly convincing. If the users are -allowed to delete all data in a given table, then that table must be -dedicated to them anyway; so it's not that easy to see why you can't -risk giving them ownership rights on it. The worst they can do is -screw up their own data, no? - -In any case, I don't see what's so wrong with the model of using -SECURITY DEFINER interface functions when you want a security -restriction that's finer-grain than the system provides. I really -*don't* want to see us trying to, say, categorize every variety of -ALTER TABLE as a separately grantable privilege. I could live with -something like a catchall "ADMIN" privilege ... except it's not -clear how that would differ from ownership. - - regards, tom lane - -From pgsql-hackers-owner+M78221=pgman=candle.pha.pa.us@postgresql.org Tue Jan 3 13:30:34 2006 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -Date: Tue, 3 Jan 2006 13:30:56 -0500 -From: Stephen Frost -To: Tom Lane -cc: Jim C. Nasby , - Bruce Momjian , - Andrew Dunstan , kleptog@svana.org, - simon@2ndquadrant.com, gsstark@mit.edu, pg@rbt.ca, zhouqq@cs.toronto.edu, - pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -Message-ID: <20060103183056.GR6026@ns.snowman.net> -Mail-Followup-To: Tom Lane , - "Jim C. Nasby" , - Bruce Momjian , - Andrew Dunstan , kleptog@svana.org, - simon@2ndquadrant.com, gsstark@mit.edu, pg@rbt.ca, - zhouqq@cs.toronto.edu, pgsql-hackers@postgresql.org -References: <200512291605.jBTG5gi00396@candle.pha.pa.us> <7966.1135873468@sss.pgh.pa.us> <20060103154521.GC82560@pervasive.com> <20060103162137.GO6026@ns.snowman.net> <16856.1136305742@sss.pgh.pa.us> <20060103165359.GP6026@ns.snowman.net> <17841.1136309852@sss.pgh.pa.us> -Content-Disposition: inline -In-Reply-To: <17841.1136309852@sss.pgh.pa.us> -X-Editor: Vim http://www.vim.org/ -X-Info: http://www.snowman.net -X-Operating-System: Linux/2.4.24ns.3.0 (i686) -X-Uptime: 12:39:16 up 206 days, 9:50, 11 users, load average: 0.02, 0.05, 0.05 -User-Agent: Mutt/1.5.9i -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.105 required=5 tests=[AWL=0.105] -X-Spam-Score: 0.105 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 2666 - --- Start of PGP signed section. -* Tom Lane (tgl@sss.pgh.pa.us) wrote: -> I don't find this use-case particularly convincing. If the users are -> allowed to delete all data in a given table, then that table must be -> dedicated to them anyway; so it's not that easy to see why you can't -> risk giving them ownership rights on it. The worst they can do is -> screw up their own data, no? - -Being able to delete all data in a given table in no way implies -ownership rights. The tables are part of a specification which the -users are being asked to respond to. Being able to change the table -types or remove the constraints put on the tables would allow the -users to upload garbage which would then affect downstream processing. - -We can't guarentee this won't happen anyway but we try to confine the -things they can mess up to a reasonable set which we can check for (and -do, through a rather involved error checking system). There are *alot* -of things built on top of the table structures and having them change -would basically break the whole system (without the appropriate changes -being made to the other parts of the system). - -> In any case, I don't see what's so wrong with the model of using -> SECURITY DEFINER interface functions when you want a security -> restriction that's finer-grain than the system provides. I really -> *don't* want to see us trying to, say, categorize every variety of -> ALTER TABLE as a separately grantable privilege. I could live with -> something like a catchall "ADMIN" privilege ... except it's not -> clear how that would differ from ownership. - -I don't think anyone's asked for 'ALTER TABLE' privileges to be -seperately grantable. It seems to me that the privileges which *need* -to be grantable are ones associated with DML statements. I would -classify TRUNCATE, VACUUM and ANALYZE as DML statements (along with -select, insert, update, and delete). They're PostgreSQL-specific DML -statements but they still fall into that category. I don't think -it's a coincidence that the SQL-defined DML statements are all, -individually, grantable. - -That doesn't mean I think we should get rid of RULE, REFERENCES or -TRIGGER, though honestly I've very rarely needed to grant any of them -(I don't think I've ever granted RULE or TRIGGER...). References is -DDL-oriented, but for *other* tables; RULE and TRIGGER are DDL and I -can't really justify why someone other than the owner would need them -but I'm guessing someone's using them. I don't think their existance -should imply that if we ever change the grants again we have to include -all types of 'ALTER TABLE', etc, though. - - Thanks, - - Stephen --- End of PGP section, PGP failed! - -From sfrost@snowman.net Tue Jan 3 13:30:13 2006 -Date: Tue, 3 Jan 2006 13:30:56 -0500 -From: Stephen Frost -To: Tom Lane -cc: Jim C. Nasby , - Bruce Momjian , - Andrew Dunstan , kleptog@svana.org, - simon@2ndquadrant.com, gsstark@mit.edu, pg@rbt.ca, zhouqq@cs.toronto.edu, - pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -Message-ID: <20060103183056.GR6026@ns.snowman.net> -Mail-Followup-To: Tom Lane , - "Jim C. Nasby" , - Bruce Momjian , - Andrew Dunstan , kleptog@svana.org, - simon@2ndquadrant.com, gsstark@mit.edu, pg@rbt.ca, - zhouqq@cs.toronto.edu, pgsql-hackers@postgresql.org -References: <200512291605.jBTG5gi00396@candle.pha.pa.us> <7966.1135873468@sss.pgh.pa.us> <20060103154521.GC82560@pervasive.com> <20060103162137.GO6026@ns.snowman.net> <16856.1136305742@sss.pgh.pa.us> <20060103165359.GP6026@ns.snowman.net> <17841.1136309852@sss.pgh.pa.us> -Content-Disposition: inline -In-Reply-To: <17841.1136309852@sss.pgh.pa.us> -X-Editor: Vim http://www.vim.org/ -X-Info: http://www.snowman.net -X-Operating-System: Linux/2.4.24ns.3.0 (i686) -X-Uptime: 12:39:16 up 206 days, 9:50, 11 users, load average: 0.02, 0.05, 0.05 -User-Agent: Mutt/1.5.9i -Content-Length: 2666 - --- Start of PGP signed section. -* Tom Lane (tgl@sss.pgh.pa.us) wrote: -> I don't find this use-case particularly convincing. If the users are -> allowed to delete all data in a given table, then that table must be -> dedicated to them anyway; so it's not that easy to see why you can't -> risk giving them ownership rights on it. The worst they can do is -> screw up their own data, no? - -Being able to delete all data in a given table in no way implies -ownership rights. The tables are part of a specification which the -users are being asked to respond to. Being able to change the table -types or remove the constraints put on the tables would allow the -users to upload garbage which would then affect downstream processing. - -We can't guarentee this won't happen anyway but we try to confine the -things they can mess up to a reasonable set which we can check for (and -do, through a rather involved error checking system). There are *alot* -of things built on top of the table structures and having them change -would basically break the whole system (without the appropriate changes -being made to the other parts of the system). - -> In any case, I don't see what's so wrong with the model of using -> SECURITY DEFINER interface functions when you want a security -> restriction that's finer-grain than the system provides. I really -> *don't* want to see us trying to, say, categorize every variety of -> ALTER TABLE as a separately grantable privilege. I could live with -> something like a catchall "ADMIN" privilege ... except it's not -> clear how that would differ from ownership. - -I don't think anyone's asked for 'ALTER TABLE' privileges to be -seperately grantable. It seems to me that the privileges which *need* -to be grantable are ones associated with DML statements. I would -classify TRUNCATE, VACUUM and ANALYZE as DML statements (along with -select, insert, update, and delete). They're PostgreSQL-specific DML -statements but they still fall into that category. I don't think -it's a coincidence that the SQL-defined DML statements are all, -individually, grantable. - -That doesn't mean I think we should get rid of RULE, REFERENCES or -TRIGGER, though honestly I've very rarely needed to grant any of them -(I don't think I've ever granted RULE or TRIGGER...). References is -DDL-oriented, but for *other* tables; RULE and TRIGGER are DDL and I -can't really justify why someone other than the owner would need them -but I'm guessing someone's using them. I don't think their existance -should imply that if we ever change the grants again we have to include -all types of 'ALTER TABLE', etc, though. - - Thanks, - - Stephen --- End of PGP section, PGP failed! - -From pgsql-hackers-owner+M78233=pgman=candle.pha.pa.us@postgresql.org Tue Jan 3 17:39:06 2006 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -From: Bruce Momjian -Message-ID: <200601032238.k03McP804163@candle.pha.pa.us> -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -In-Reply-To: <20060103212750.GT82560@pervasive.com> -To: Jim C. Nasby -Date: Tue, 3 Jan 2006 17:38:25 -0500 (EST) -cc: Tom Lane , Andrew Dunstan , - simon@2ndquadrant.com, kleptog@svana.org, gsstark@mit.edu, pg@rbt.ca, - zhouqq@cs.toronto.edu, pgsql-hackers@postgresql.org -X-Mailer: ELM [version 2.4ME+ PL121 (25)] -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.121 required=5 tests=[AWL=0.121] -X-Spam-Score: 0.121 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 1714 - -Jim C. Nasby wrote: -> > We would be creating a new lock type for this. -> -> Sorry if I've just missed this in the thread, but what would the new -> lock type do? My impression is that as it stands you can either do: -> -> BEGIN; -> ALTER TABLE EXCLUSIVE; -> ... -> ALTER TABLE SHARE; --fsync -> COMMIT; -> -> Which would block all other access to the table as soon as the first -> ALTER TABLE happens. Or you can: -> -> ALTER TABLE EXCLUSIVE; -> ... -> ALTER TABLE SHARE; -> -> Which means that between the two ALTER TABLES every backend that does -> DML on that table will not have that DML logged, but because there's no -> exclusive lock that DML would be allowed to occur. - -Right, the DML will be single-threaded and fsync of all dirty pages will -happen before commit of each transaction. - -> BTW, there might be some usecase for the second scenario, in which case -> it would probably be better to tell the user to aquire a table-lock on -> their own rather than do it automatically as part of the update... - -> > Basically meaning your idea of update while EXCLUSIVE/PRESERVE/STABLE is -> > happening is never going to be implemented because it is just too hard -> > to do, and too prone to error. -> -> What I figured. Never hurts to ask though. :) - -Actually, it does hurt because it generates discussion volume for no -purpose. - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 359-1001 - + If your life is a hard drive, | 13 Roberts Road - + Christ can be your backup. | Newtown Square, Pennsylvania 19073 - ----------------------------(end of broadcast)--------------------------- -TIP 5: don't forget to increase your free space map settings - -From pgsql-hackers-owner+M78234=pgman=candle.pha.pa.us@postgresql.org Tue Jan 3 17:54:16 2006 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -From: Simon Riggs -To: Bruce Momjian -cc: Andrew Dunstan , Tom Lane , - Martijn van Oosterhout , Greg Stark , - Rod Taylor , Qingqing Zhou , - pgsql-hackers@postgresql.org -In-Reply-To: <200512302114.jBULEno02301@candle.pha.pa.us> -References: <200512302114.jBULEno02301@candle.pha.pa.us> -Date: Tue, 03 Jan 2006 22:53:53 +0000 -Message-ID: <1136328833.5052.223.camel@localhost.localdomain> -X-Mailer: Evolution 2.2.3 (2.2.3-2.fc4) -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.04 required=5 tests=[AWL=0.040] -X-Spam-Score: 0.04 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 5373 - -On Fri, 2005-12-30 at 16:14 -0500, Bruce Momjian wrote: -> Simon Riggs wrote: -> > The implications of EXCLUSIVE are: -> > - there will be a check on each and every I, U, D to check the state of -> > the relation -> > - *every* operation that attempts a write lock will attempt to acquire -> > an EXCLUSIVE full table lock instead -> > - following successful completion of *each* DML statement, the relation -> > will be heap_sync'd involving a full scan of the buffer cache -> -> Yes, I think that is it. What we can do is implement EXCLUSIVE to -> affect only COPY at this point, and document that, and later add other -> commands. -> -> > Can I clarify the wording of the syntax? Is EXCLUSIVE the right word? -> > How about FASTLOAD or BULKLOAD? Those words seem less likely to be -> > misused in the future - i.e. we are invoking a special mode, rather than -> > invoking a special "go faster" option. -> -> The problem with the FASTLOAD/BULKLOAD words is that EXCLUSIVE mode is -> probably not the best for loading. I would think TRUNCATE would be a -> better option. -> -> In fact, in loading a table, I think both EXCLUSIVE and TRUNCATE would be -> the same, mostly. You would create the table, set its RELIABILITY to -> TRUNCATE, COPY into the table, then set the RELIABILITY to SHARE or -> DEFAULT. The second ALTER has to sync all the dirty data blocks, which -> the same thing EXCLUSIVE does at the conclusion of COPY. -> -> So, we need a name for EXCLUSIVE mode that suggests how it is different -> from TRUNCATE, and in this case, the difference is that EXCLUSIVE -> preserves the previous contents of the table on recovery, while TRUNCATE -> does not. Do you want to call the mode PRESERVE, or EXCLUSIVE WRITER? -> Anyway, the keywords are easy to modify, even after the patch is -> submitted. FYI, I usually go through keywords.c looking for a keyword -> we already use. - -I'm very happy for suggestions on what these new modes are called. - -> > > So, to summarize, I think we should add DROP/TRUNCATE, and use that by -> > > default (or optionally off?) in pg_dump, and, assuming we want EXCLUSIVE -> > > for more than just COPY, we need to add ALTER TABLE EXCLUSIVE. -> > -> > Would you mind stating again what you mean, just so I can understand -> > this? Your summary isn't enough. -> -> New ALTER TABLE mode, perhaps call it PERSISTENCE: -> -> ALTER TABLE tab PERSISTENCE DROP ON RECOVERY -> ALTER TABLE tab PERSISTENCE TRUNCATE ON RECOVERY -> -> These would drop or truncate all tables with this flag on a non-clean -> start of the postmaster, and write something in the server logs. -> However, I don't know that we have the code in place to DROP/TRUNCATE in -> recovery mode, and it would affect all databases, so it could be quite -> complex to implement. In this mode, no WAL logs would be written for -> table modifications, though DDL commands would have to be logged. - -Right now, this will be a TODO item... it looks like it will take some -thought to implement correctly. - -> ALTER TABLE tab PERSISTENCE PRESERVE (or STABLE?) -> -> Table contents are preserved across recoveries, but data modifications -> can happen only one at a time. I don't think we have a lock mode that -> does this, so I am worried a new lock mode will have to be created. A -> simplified solution at this stage would be to take an exclusive lock on -> the table, but really we just need a single-writer table lock, which I -> don't think we have. initially this can implemented to only affect COPY -> but later can be done for other commands. - -ExclusiveLock locks out everything apart from readers, no new lock mode -AFAICS. Implementing that is little additional work for COPY. - -Tom had a concern about setting this for I, U, D commands via the -executor. Not sure what the details of that are, as yet. - -We can use either of the unlogged modes for pg_dump, so I'd suggest its -this one. Everybody happy with this being the new default in pg_dump, or -should it be an option? - -> ALTER TABLE tab PERSISTENCE DEFAULT -> -> This would be our current default mode, which is full concurrency and -> persistence. - -I'm thinking whether the ALTER TABLE statement might be better with two -bool flags rather than a 3-state char. - -flag 1: ENABLE LOGGING | DISABLE LOGGING - -flag 2: FULL RECOVERY | TRUNCATE ON RECOVERY - -Giving 3 possible sets of options: - --- the default -ALTER TABLE mytable ENABLE LOGGING FULL RECOVERY; (default) - --- EXCLUSIVE mode -ALTER TABLE mytable DISABLE LOGGING FULL RECOVERY; -...which would be used like this - ALTER TABLE mytable DISABLE LOGGING; - COPY or other bulk data manipulation SQL - ALTER TABLE mytable ENABLE LOGGING; -...since FULL RECOVERY is the default. - --- multiuser temp table mode -ALTER TABLE mytable DISABLE LOGGING TRUNCATE ON RECOVERY; -...which would usually be left on all the time - -which only uses one new keyword LOGGING and yet all the modes are fairly -explicit as to what they do. - -An alternative might be the slightly more verbose: - ALTER TABLE mytable DISABLE LOGGING FORCE EXCLUSIVE TABLE LOCK; -which would be turned off by - ALTER TABLE mytable ENABLE LOGGING; - -Comments? - -Best Regards, Simon Riggs - - ----------------------------(end of broadcast)--------------------------- -TIP 1: if posting/reading through Usenet, please send an appropriate - subscribe-nomail command to majordomo@postgresql.org so that your - message can get through to the mailing list cleanly - -From simon@2ndquadrant.com Tue Jan 3 18:10:32 2006 -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -From: Simon Riggs -To: Jim C. Nasby , - Bruce Momjian -cc: Tom Lane , Andrew Dunstan , - kleptog@svana.org, gsstark@mit.edu, pg@rbt.ca, zhouqq@cs.toronto.edu, - pgsql-hackers@postgresql.org -In-Reply-To: <200601032120.k03LKl609990@candle.pha.pa.us> -References: <200601032120.k03LKl609990@candle.pha.pa.us> -Date: Tue, 03 Jan 2006 23:10:16 +0000 -Message-ID: <1136329816.5052.239.camel@localhost.localdomain> -X-Mailer: Evolution 2.2.3 (2.2.3-2.fc4) -Content-Length: 2118 - -On Tue, 2006-01-03 at 16:20 -0500, Bruce Momjian wrote: -> Jim C. Nasby wrote: - -> > Idealistically, if EXCLUSIVE/PRESERVE/STABLE does it's thing by only -> > appending new pages, it would be nice if other backends could continue -> > performing updates at the same time, assuming there's free space -> > available elsewhere within the table (and that you'd be able to recover -> > those logged changes regardless of the non-logged operations). But -> > that's a pretty lofty goal... -> -> "Idealistically", yep. It would be great if we could put a helmet on -> and the computer would read your mind. :-) -> -> Basically meaning your idea of update while EXCLUSIVE/PRESERVE/STABLE is -> happening is never going to be implemented because it is just too hard -> to do, and too prone to error. - -The reason for locking the whole table was to ensure that we do not have -a mixture of logged and non-logged writers writing to the same data -blocks, since that could damage blocks unrecoverably in the event of a -crash. (Though perhaps only if full_block_writes is on) - -The ALTER TABLE .. EXCLUSIVE/(insert name) mode would mean that *any* -backend who took a write lock on the table, would lock out the whole -table. So this new mode is not restricted to the job/user who ran the -ALTER TABLE command. (I would note that that is how Oracle and Teradata -do this for pre-load utility table locking, but why should we follow -them on that?) - -Currently, when we add a new row when the FSM is empty, we check the -last block of the table. That would cause multiple writers to access the -same blocks and so we would be in danger. The only way to avoid that -would be for logged writers (who would use the FSM if it were not empty) -to notify back to the FSM that they have just added a block - and remove -the behaviour to look for the last block. - -Anyway, one step at a time. *Maybe* we can do that in the future, but -right now I'd like to add the basic fast write/load functionality. - -Also, I think I will do the docs first this time, just so everyone can -read what we're getting ahead of time, to ensure we all agree. - -Best Regards, Simon Riggs - -From pgsql-hackers-owner+M78236=pgman=candle.pha.pa.us@postgresql.org Tue Jan 3 18:24:20 2006 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -From: Simon Riggs -To: Bruce Momjian -cc: Jim C. Nasby , Tom Lane , - Andrew Dunstan , kleptog@svana.org, gsstark@mit.edu, - pg@rbt.ca, zhouqq@cs.toronto.edu, pgsql-hackers@postgresql.org -In-Reply-To: <200601032238.k03McP804163@candle.pha.pa.us> -References: <200601032238.k03McP804163@candle.pha.pa.us> -Date: Tue, 03 Jan 2006 23:23:54 +0000 -Message-ID: <1136330634.5052.247.camel@localhost.localdomain> -X-Mailer: Evolution 2.2.3 (2.2.3-2.fc4) -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.043 required=5 tests=[AWL=0.043] -X-Spam-Score: 0.043 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 725 - -On Tue, 2006-01-03 at 17:38 -0500, Bruce Momjian wrote: - -> Right, the DML will be single-threaded and fsync of all dirty pages will -> happen before commit of each transaction. - -heap_sync() would occur at end of statement, as it does with CTAS. We -could delay until EOT but I'm not sure I see why; in most cases they'd -be the same point anyway. - -I'd been toying with the idea of making the freshly added blocks live -only in temp_buffers to avoid the shared_buffers overhead, but that was -starting to sounds too wierd for my liking. - -Best Regards, Simon Riggs - - ----------------------------(end of broadcast)--------------------------- -TIP 4: Have you searched our list archives? - - http://archives.postgresql.org - -From simon@2ndquadrant.com Tue Jan 3 18:58:13 2006 -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -From: Simon Riggs -To: Michael Paesold -cc: Bruce Momjian , andrew@dunslane.net, - tgl@sss.pgh.pa.us, kleptog@svana.org, gsstark@mit.edu, pg@rbt.ca, - zhouqq@cs.toronto.edu, pgsql-hackers@postgresql.org -In-Reply-To: <14969.1136030384@www6.gmx.net> -References: <200512302114.jBULEno02301@candle.pha.pa.us> - <14969.1136030384@www6.gmx.net> -Date: Tue, 03 Jan 2006 23:58:09 +0000 -Message-ID: <1136332689.5052.263.camel@localhost.localdomain> -X-Mailer: Evolution 2.2.3 (2.2.3-2.fc4) -Content-Length: 1493 - -On Sat, 2005-12-31 at 12:59 +0100, Michael Paesold wrote: -> Bruce Momjian wrote: -> -> > > The --single-transaction mode would apply even if the dump was created -> > > using an earlier version of pg_dump. pg_dump has *not* been altered at -> > > all. (And I would again add that the idea was not my own) -> > -> > I assume you mean this: -> > -> > http://archives.postgresql.org/pgsql-patches/2005-12/msg00257.php -> > -> > I guess with the ALTER commands I don't see much value in the -> > --single-transaction flag. I am sure others suggested it, but would -> > they suggest it now given our current direction. -> -> I just want to add that --single-transaction has a value of it's own. There -> were times when I wanted to restore parts of a dump all-or-nothing. -> -> This is possible with PostgreSQL, unlike many other DBM systems, because -> people like Tom Lane have invested in ensuring that all DDL is working -> without implicitly committing an enclosing transaction. -> -> Using pg_restore directly into a database, it is not possible to get a -> single transaction right now. One has to restore to a file and manually -> added BEGIN/COMMIT. Just for that I think --single-transaction is a great -> addition and a missing feature. -> -> I think more people have a use-case for that. - -I did originally separate the --single-transaction patch for this -reason. I think its a valid patch on its own and its wrapped and ready -to go, with some deletions from the doc patch. - -Best Regards, Simon Riggs - -From pgsql-hackers-owner+M78239=pgman=candle.pha.pa.us@postgresql.org Tue Jan 3 19:12:18 2006 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -From: Simon Riggs -To: Bruce Momjian -cc: Tom Lane , Martijn van Oosterhout , - Greg Stark , Rod Taylor , - Qingqing Zhou , pgsql-hackers@postgresql.org -In-Reply-To: <200512291637.jBTGbdC03848@candle.pha.pa.us> -References: <200512291637.jBTGbdC03848@candle.pha.pa.us> -Date: Wed, 04 Jan 2006 00:11:55 +0000 -Message-ID: <1136333515.5052.273.camel@localhost.localdomain> -X-Mailer: Evolution 2.2.3 (2.2.3-2.fc4) -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.045 required=5 tests=[AWL=0.045] -X-Spam-Score: 0.045 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 1200 - -On Thu, 2005-12-29 at 11:37 -0500, Bruce Momjian wrote: -> Having COPY behave differently because it is -> in a transaction is fine as long as it is user-invisible, but once you -> require users to do that to get the speedup, it isn't user-invisible -> anymore. - -Since we're agreed on adding ALTER TABLE rather than COPY LOCK, we have -our explicit mechanism for speedup. - -However, it costs a single line of code and very very little execution -time to add in the optimization to COPY to make it bypass WAL when -executed in the same transaction that created the table. Everything else -is already there. - -As part of the use_wal test: -+ if (resultRelInfo->ri_NumIndices == 0 && -+ !XLogArchivingActive() && ->> (cstate->rel->rd_createSubid != InvalidSubTransactionId )) -+ use_wal = false; - -the value is already retrieved from cache... - -Can anyone see a reason *not* to put that change in also? We just don't -advertise it as the "suggested" route to gaining performance, nor would -we rely on it for pg_dump/restore performance. - -Best Regards, Simon Riggs - - ----------------------------(end of broadcast)--------------------------- -TIP 2: Don't 'kill -9' the postmaster - -From pgsql-hackers-owner+M78303=pgman=candle.pha.pa.us@postgresql.org Thu Jan 5 12:23:39 2006 -X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org -X-Greylist: from auto-whitelisted by SQLgrey- -From: Bruce Momjian -Message-ID: <200601051722.k05HMSM02052@candle.pha.pa.us> -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -In-Reply-To: <1136328833.5052.223.camel@localhost.localdomain> -To: Simon Riggs -Date: Thu, 5 Jan 2006 12:22:28 -0500 (EST) -cc: Andrew Dunstan , Tom Lane , - Martijn van Oosterhout , Greg Stark , - Rod Taylor , Qingqing Zhou , - pgsql-hackers@postgresql.org -X-Mailer: ELM [version 2.4ME+ PL121 (25)] -X-Virus-Scanned: by amavisd-new at hub.org -X-Spam-Status: No, score=0.12 required=5 tests=[AWL=0.120] -X-Spam-Score: 0.12 -X-Spam-Level: -X-Mailing-List: pgsql-hackers -List-Archive: -List-Help: -List-Id: -List-Owner: -List-Post: -List-Subscribe: -List-Unsubscribe: -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Content-Length: 6020 - -Simon Riggs wrote: -> > So, we need a name for EXCLUSIVE mode that suggests how it is different -> > from TRUNCATE, and in this case, the difference is that EXCLUSIVE -> > preserves the previous contents of the table on recovery, while TRUNCATE -> > does not. Do you want to call the mode PRESERVE, or EXCLUSIVE WRITER? -> > Anyway, the keywords are easy to modify, even after the patch is -> > submitted. FYI, I usually go through keywords.c looking for a keyword -> > we already use. -> -> I'm very happy for suggestions on what these new modes are called. -> -> > > > So, to summarize, I think we should add DROP/TRUNCATE, and use that by -> > > > default (or optionally off?) in pg_dump, and, assuming we want EXCLUSIVE -> > > > for more than just COPY, we need to add ALTER TABLE EXCLUSIVE. -> > > -> > > Would you mind stating again what you mean, just so I can understand -> > > this? Your summary isn't enough. -> > -> > New ALTER TABLE mode, perhaps call it PERSISTENCE: -> > -> > ALTER TABLE tab PERSISTENCE DROP ON RECOVERY -> > ALTER TABLE tab PERSISTENCE TRUNCATE ON RECOVERY -> > -> > These would drop or truncate all tables with this flag on a non-clean -> > start of the postmaster, and write something in the server logs. -> > However, I don't know that we have the code in place to DROP/TRUNCATE in -> > recovery mode, and it would affect all databases, so it could be quite -> > complex to implement. In this mode, no WAL logs would be written for -> > table modifications, though DDL commands would have to be logged. -> -> Right now, this will be a TODO item... it looks like it will take some -> thought to implement correctly. - -OK, I know my suggestions have made it more complicated. - -TODO added: - -* Allow control over which tables are WAL-logged - - Allow tables to bypass WAL writes and just fsync() dirty pages on - commit. To do this, only a single writer can modify the table, and - writes must happen only on new pages. Readers can continue accessing - the table. This would affect COPY, and perhaps INSERT/UPDATE too. - Another option is to avoid transaction logging entirely and truncate - or drop the table on crash recovery. These should be implemented - using ALTER TABLE, e.g. ALTER TABLE PERSISTENCE [ DROP | TRUNCATE | - STABLE | DEFAULT ]. Tables using non-default logging should not use - referential integrity with default-logging tables, and tables using - stable logging probably can not have indexes. [walcontrol] - - -> > ALTER TABLE tab PERSISTENCE PRESERVE (or STABLE?) -> > -> > Table contents are preserved across recoveries, but data modifications -> > can happen only one at a time. I don't think we have a lock mode that -> > does this, so I am worried a new lock mode will have to be created. A -> > simplified solution at this stage would be to take an exclusive lock on -> > the table, but really we just need a single-writer table lock, which I -> > don't think we have. initially this can implemented to only affect COPY -> > but later can be done for other commands. -> -> ExclusiveLock locks out everything apart from readers, no new lock mode -> AFAICS. Implementing that is little additional work for COPY. - -Nice. - -> Tom had a concern about setting this for I, U, D commands via the -> executor. Not sure what the details of that are, as yet. - -That is much more complicated than the COPY-only idea, for sure. I am -thinking we could add the ALTER syntax and just do COPY at this stage, -meaning that I/U/D still do full logging until we get to improving them. -The big benefit is that the user API doesn't need to change when we -improve the code. In fact I think we could do the TRUNCATE/DROP easily -for I/U/D, but the STABLE option would require work and we don't need to -implement it in the first patch. - -> We can use either of the unlogged modes for pg_dump, so I'd suggest its -> this one. Everybody happy with this being the new default in pg_dump, or -> should it be an option? -> -> > ALTER TABLE tab PERSISTENCE DEFAULT -> > -> > This would be our current default mode, which is full concurrency and -> > persistence. -> -> I'm thinking whether the ALTER TABLE statement might be better with two -> bool flags rather than a 3-state char. -> -> flag 1: ENABLE LOGGING | DISABLE LOGGING -> -> flag 2: FULL RECOVERY | TRUNCATE ON RECOVERY -> -> Giving 3 possible sets of options: -> -> -- the default -> ALTER TABLE mytable ENABLE LOGGING FULL RECOVERY; (default) -> -> -- EXCLUSIVE mode -> ALTER TABLE mytable DISABLE LOGGING FULL RECOVERY; -> ...which would be used like this -> ALTER TABLE mytable DISABLE LOGGING; -> COPY or other bulk data manipulation SQL -> ALTER TABLE mytable ENABLE LOGGING; -> ...since FULL RECOVERY is the default. -> -> -- multiuser temp table mode -> ALTER TABLE mytable DISABLE LOGGING TRUNCATE ON RECOVERY; -> ...which would usually be left on all the time -> -> which only uses one new keyword LOGGING and yet all the modes are fairly -> explicit as to what they do. -> -> An alternative might be the slightly more verbose: -> ALTER TABLE mytable DISABLE LOGGING FORCE EXCLUSIVE TABLE LOCK; -> which would be turned off by -> ALTER TABLE mytable ENABLE LOGGING; -> -> Comments? - -I had the same idea originally, but avoided it because the logging -really does affect what other options you can use. For example, if you -want truncate on recovery, you certainly do not want logging, so it -seems the options are not really independent. In fact if someone asks -for truncate on recovery, do we automatically turn off logging for them, -or throw an error, or a warning. It just seemed too error-prone and -confusing, though perhaps more logical. Of course, if others like the -above, we can do it. - --- - Bruce Momjian | http://candle.pha.pa.us - pgman@candle.pha.pa.us | (610) 359-1001 - + If your life is a hard drive, | 13 Roberts Road - + Christ can be your backup. | Newtown Square, Pennsylvania 19073 - ----------------------------(end of broadcast)--------------------------- -TIP 2: Don't 'kill -9' the postmaster - -From simon@2ndquadrant.com Thu Jan 5 16:56:25 2006 -Return-path: -Received: from smtp.nildram.co.uk (smtp.nildram.co.uk [195.112.4.54]) - by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id k05LuPb02246 - for ; Thu, 5 Jan 2006 16:56:25 -0500 (EST) -Received: from [192.168.0.3] (unknown [84.12.184.6]) - by smtp.nildram.co.uk (Postfix) with ESMTP - id A9F0F268C4E; Thu, 5 Jan 2006 21:56:18 +0000 (GMT) -Subject: Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and -From: Simon Riggs -To: Bruce Momjian -cc: Tom Lane , Martijn van Oosterhout , - Greg Stark , Rod Taylor , - Qingqing Zhou , pgsql-hackers@postgresql.org -In-Reply-To: <200601051727.k05HR5p02803@candle.pha.pa.us> -References: <200601051727.k05HR5p02803@candle.pha.pa.us> -Content-Type: text/plain -Date: Thu, 05 Jan 2006 21:56:21 +0000 -Message-ID: <1136498181.21025.285.camel@localhost.localdomain> -MIME-Version: 1.0 -X-Mailer: Evolution 2.2.3 (2.2.3-2.fc4) -Content-Transfer-Encoding: 7bit -Status: OR - -On Thu, 2006-01-05 at 12:27 -0500, Bruce Momjian wrote: - -> Seems like a nice optimization. - -Negative thoughts: Toast tables have a toast index on them, yes? We have -agreed that we cannot use the optimization if we have indexes on the -main table. It follows that we cannot use the optimization if we have -*any* toasted data, since that would require a pointer between two -blocks, which would not be correctly recovered following a crash. If we -log the toast table then there could be a mismatch between heap and -toast table; if we don't log the toast table there could be a mismatch -between toast table and toast index. - -We can test to see if the toast table is empty when we do ALTER TABLE, -but loading operations may try to create toasted data rows. - -Presumably that means we must either: -i) abort a COPY if we get a toastable value -ii) if we get a toastable value, insert the row into a new block, which -we do logging of, then also log the toast insert and the toast index -insert - i.e. some blocks we log, others not - -This is still useful for many applications, IMHO, but the list of -restrictions seems to be growing. Worse, we wouldn't know that the toast -tables were empty until after we did the COPY TO for a pg_dump, so we -wouldn't be able to retrospectively add an ALTER TABLE command ahead of -the COPY. - -Thoughts? Hopefully there are some flaws in my thinking here, - -Best Regards, Simon Riggs - - -