postgres/doc/TODO

TODO list for PostgreSQL
========================
Last updated:		Sat Mar 21 00:26:29 EST 1998

Current maintainer:	Bruce Momjian (maillist@candle.pha.pa.us)

The most recent version of this document can be viewed at
the PostgreSQL WWW site, http://www.postgreSQL.org.

THE CHANGES FOR 6.3 APPEAR AT THE END OF THIS DOCUMENT

A dash(-) marks changes to be in the next release.

Developers who have claimed items are:
-------------------------------------
	* Bruce is Bruce Momjian<maillist@candle.pha.pa.us>
	* Bryan is Bryan Henderson<bryanh@giraffe.netgate.net>
	* D'Arcy is D'Arcy J.M. Cain <darcy@druid.net>
	* Dan is Dan McGuirk <mcguirk@indirect.com>
	* Daniel is Daniel Kalchev <daniel@digsys.bg>
	* Darren is Darren King <darrenk@insightdist.com>
	* Edmund is Edmund Mergl <E.Mergl@bawue.de>
	* Erich Stamberger <eberger@gewi.kfunigraz.ac.at>
	* Gerhard is Gerhard Reithofer <gerhardr@tech-edv.co.at>
	* Goran is Goran Thyni <goran@bildbasen.se>
	* Henry is Henry B. Hotz <hotz@jpl.nasa.gov>
	* Igor is Igor <igor@sba.miami.edu>
	* Jan is Jan Wieck <wieck@sapserv.debis.de>
	* Jun is Jun Kuwamura <juk@rccm.co.jp>
	* Kurt is "Kurt J. Lidl" <lidl@va.pubnix.com>
 	* Marc is Marc Fournier <scrappy@hub.org>
 	* Martin is Martin S. Utesch <utesch@aut.tu-freiberg.de>
	* Michael is Michael Meskes <meskes@debian.org>
	* Oleg is Oleg Bartunov <oleg@sai.msu.su>
	* Paul is Paul M. Aoki <aoki@CS.Berkeley.EDU>
	* Patrick is Patrick van Kleef <pvk@pobox.com>
	* Peter is Peter T Mount <psqlhack@maidast.demon.co.uk>
	* Phil is Phil Thompson <phil@river-bank.demon.co.uk>
	* Raymond is Raymond Toy <toy@rtp.ericsson.se>
	* Soo-Ho Ok <shok@detc.dongeui-tc.ac.kr>
	* Stefan Simkovics <ssimkovi@rainbow.studorg.tuwien.ac.at>
	* Sven is Sven Verdoolaege <skimo@breughel.ufsia.ac.be>
	* Tatsuo is Tatsuo Ishii <t-ishii@sra.co.jp>
	* Thomas is Thomas Lockhart <tgl@mythos.jpl.nasa.gov>
	* Todd is Todd Brandys is <brandys@eng3.hep.uiuc.edu>
	* Vadim is "Vadim B. Mikheev" <vadim@sable.krasnoyarsk.su>
	* Vivek is Vivek Khera <khera@kci.kciLink.com>

Additional 6.3 developers include:
---------------------------------
	* Billy is Billy G. Allie <Bill.Allie@mug.org>
	* Brook is Brook Milligan <brook@trillium.NMSU.Edu>
	* James is James Hughes <jamesh@interpath.com>
	* Jeroen is Jeroen van Vianen <jeroenv@design.nl>
	* Matt is Matt Maycock <maycock@intelliquest.com>
	* Ryan is Ryan Kirkpatrick <rkirkpat@nag.cs.colorado.edu>
	* TomH is Tom I Helbekkmo <tih@Hamartun.Priv.NO>
	* TomS is Tom Szybist
	* Travis is Travis Melhiser <melhiser@viper.co.union.nc.us>

RELIABILITY
-----------
* Overhaul mdmgr/smgr to fix double unlinking and double opens, cleanup
* Overhaul bufmgr/lockmgr/transaction manager
* Remove EXTEND?
* CREATE VIEW requires super-user priviledge
* Can lo_export()/lo_import() read/write anywhere, causing a security problem?
* Tables that start with xinv confused to be large objects
* Two and three dimmensional arrays display improperly, missing {}
* GROUP BY in INSERT INTO table SELECT * FROM table2 fails
* Prevent auto-table reference, like SELECT table.col WHERE col = 3 (?)
* SELECT * FROM table WHERE int4_column = '1' fails
* SELECT a[1] FROM test fails, it needs test.a[1]
* UPDATE table SET table.value = 3 fails
* Make pg_dump preserve inheritance column order, do non-inherits first
* User who can create databases can modify pg_database table
* optimizer memory exhaustion with many OR's
* elog() does not free all its memory(Jan)
* views on subselects fail
* disallow inherited columns with the same name as new columns
* recover or force failure when disk space is exhausted

ENHANCEMENTS
------------
* Replace table-level locking with row or page-level locking(Vadim)
* Add SERIAL type
* Transaction log, so re-do log can be on a separate disk
* Allow transaction commits with rollback with no-fsync performance
* More access control over who can create tables and access the database
* Add full ANSI SQL capabilities
	* Implement HAVING clause(Stephan)
	* add OUTER joins, left and right (Thomas)
	* make VIEWs updateable where possible
	* add INTERSECTS, SUBTRACTS(Stephan)
	* add temporary tables(Bruce)
	* add sql3 recursive unions
	* add the concept of dataspaces
	* add DECIMAL, NUMERIC, DOUBLE PRECISION, BIT, BIT VARYING
 	* NCHAR (as distinguished from ordinary varchar),
	* DOMAIN capability
* Allow compression of large fields or a compressed field type
* Fix the rules system(Jan?,Soo-Ho)
	* robust
	* making INSTEAD rules work
	* add CONSTRAINT
* Full set of text operations and functions
	* word searches, concat,max() on text, char
* Large objects
	* Fix large object mapping scheme, own reltype
	* not to stuff everything as files in a single directory
	* Fix large object memory leaks
* Better interface for adding to pg_group
* Make MONEY/DECIMAL have a defined precision
* Fix tables >2G, or report error when 2G size reached
	(fix lseek()/off_t, mdextend()/RELSEG_SIZE)
* Threaded version of the server or libpq
* Allow libpq to cancel query requests
* Add REGEX internationalization
* allow row re-use without vacuum, maybe?(Vadim)
* Remove restriction that ORDER BY field must be in SELECT list
* Add word index for text fields, maybe with trigrams, i.e.:
    * ' (cat | dog) & ! fox ' meaning text has cat or dog, but not fox
* Populate backend status area and write program to dump status data
* Add ALTER TABLE DROP COLUMN feature
* Allow INSERT INTO ... SELECT to convert column types
* Add syslog functionality(Marc)
* Add STDDEV/VARIANCE() function for standard deviation computation/variance
* add UNIQUE capability to non-btree indexes
* make number of backends a config parameter, storage/sinvaladt.h:MaxBackendId
* certain indexes will not shrink, i.e. oid indexes with many inserts
* make NULL's come out at the beginning or end depending on the ORDER BY direction
* change the library/backend interface to use network byte order
* Restore unused oid's on backend exit if no one else has gotten oids
* remove non-standard types from the system, and make them loadable
* have UPDATE/DELETE clean out indexes
* allow WHERE restriction on ctid
* allow pg_descriptions when creating types, tables, columns, and functions
* Fix compile and security of Kerberos/GSSAPI code
* Allow psql to print nulls as distinct from ""(?)
* Allow variable casts with BETWEEN 'today'::asbtime AND 'today'::abstime
* Allow INSERT INTO ... SELECT ... FROM view to work
* Make VACUUM on database not lock pg_class
* Make VACUUM ANALYZE only use a readlock
* Allow cursors to be DECLAREd/OPENed/CLOSEed outside transactions
* Allow installation data block size and max tuple size configuration(Darren)
* Allow views on a UNION
* Allow views of aggregate columns
* Allow variable block sizes(Darren)
* System tables are now more update-able from SQL(Jan)
* New pg_shadow file, pg_user is now a view of pg_shadow(Jan)
* Allow flag to control COPY input/output of NULLs
* Allow CLUSTER on all tables at once, and improve CLUSTER
* Change all references of Postgres to PostgreSQL, including binary names
* Add ELOG_TIMESTAMPS to elog()(?)
* Change LOCK tablename to LOCK TABLE tablename(?)
* Allow max tuple length to be changed(Darren)
* Have psql with no database name not connect to username as default(?)
* Allow subqueries in target list
* Granting permissions to another user looses permissions for the owner

PERFORMANCE
-----------
* Use indexes in ORDER BY, min(), max()(Costin Oproiu)
* Optimizing disjunctive queries
* Fix bushy-plans (Martin)
* Other optimizer bugs
* Prevent fsync in SELECT-only queries
* Cache most recent query plan(s?)
* Shared catalog cache, reduce lseek()'s by caching table size in shared area
* Allow compression of log and meta data
* Add FILLFACTOR to index creation
* Allow indexes to be used with OR clauses(Vadim)
* update pg_statistic table to remove operator column
* make index creation use psort code, because it is now faster(Vadim)
* remove fork()/exec() of backend and make it just fork()
* Add base table name to \d index
* Allow char() not to use variable-sized header to reduce disk size
* Do async I/O to do better read-ahead of data
* Fix optmizer problem with self-table joins
* Fix memory exhaustion when using many OR's
* Use spin locks only on multi-CPU systems, yield CPU instead
* Get faster regex() code from Henry Spencer <henry@zoo.utoronto.ca>
	when it is available

DOCUMENTATION
-------------
* Update usermanual source(many)
* added features used in grammer but not in docs, like :: and CAST
* update libpq++ and pginterface manual page
* Add keyword list to documentation, already in /tools
* Add 'man pgsql' to show all manual page names



=============================================================================

CHANGES IN THE 6.3p1 PATCH 
--------------------------
ecpg cleanup/fixes, now version 1.1(Michael Meskes)
pg_user cleanup(Bruce)
large object fix for pg_dump and tclsh(alvin@camberlo.demon.co.uk)
LIKE fix for multiple adjacent underscores
fix for redefining builtin functions(Thomas)
ultrix4 cleanup
upgrade to pg_access 0.83
updated CLUSTER manual page
multi-byte character set support, see doc/README.mb(Tatsuo)
configure --with-pgport fix
pg_ident fix
big-endian fix for backend communications(Tatsuo)
SUBSTR() and substring() fix(Thomas)
several jdbc fixes(Peter)
libpgtcl improvements, see libptcl/README(Randy Kunkee)
Fix for "Datasize = 0" error(Vadim)
Prevent \do from wrapping(Bruce)
Remove duplicate Russian character set entries
Sunos4 cleanup
Allow optional TABLE keyword in LOCK and SELECT INTO(Thomas)
CREATE SEQUENCE options to allow a negative integer(Thomas)
Add "PASSWORD" as an allowed column identifier(Thomas)
Add checks for UNION target fields(Bruce)
Fix Alpha port(Dwayne Bailey)
Fix for text arrays containing quotes(Doug Gibson)
Solaris compile fix(Albert Chin-A-Young)
Better identify tcl and tk libs and includes(Bruce)



CHANGES IN THE 6.3 RELEASE
--------------------------

There are some general 6.3 issues that I want to mention.  These are
only the big items that can not be described in one sentence.

First, we now have subselects.  Now that we have them, I would like to
mention that without subselects, SQL is a very limited language.
Subselects are a major feature, and you should review your code for
places where subselects provide a better solution for your queries.  I
think you will find that there are more uses for subselects than you may
think.  Vadim has put us on the big SQL map with subselects, and fully
functional ones too.  The only thing you can't do with subselects is to
use them in the target list.

Second, 6.3 uses unix domain sockets rather than TCP/IP by default.  To
enable connections from other machines, you have to use the new
postmaster -i option, and of course edit pg_hba.conf.  Also, for this
reason, the format of pg_hba.conf has changed.

Third, char() fields will now allow faster access than varchar() or
text. Specifically, the text and varchar() have a penalty for access to
any columns after the first column of this type.  char() used to also
have this access penalty, but it no longer does.  This may suggest that
you redesign some of your tables, especially if you have short character
columns that you have defined as varchar() or text.  This and other
changes make 6.3 even faster than earlier releases.

We now have passwords definable independent of any Unix file.  There are
new SQL USER commands.  See the pg_hba.conf manual page for more
information.  There is a new table, pg_shadow, which is used to store
user information and user passwords, and it by default only SELECT-able
by the postgres super-user.  pg_user is now a view of pg_shadow, and is
SELECT-able by PUBLIC.  You should keep using pg_user in your
application without changes.

User-created tables now no longer have SELECT permission to PUBLIC by
default.  This was done because the ANSI standard requires it.  You can
of course GRANT any permissions you want after the table is created. 
System tables continue to be SELECT-able by PUBLIC.

We also have real deadlock detection code.  No more sixty-second
timeouts.  And the new locking code implements a FIFO better, so there
should be less resource starvation during heavy use.

Many complaints have been made about inadequate documenation in previous
releases.  Thomas has put much effort into many new manuals for this
release.  Check out the /doc directory.

For performance reasons, time travel is gone, but can be implemented
using triggers (see pgsql/contrib/spi/README).  Please check out the new
\d command for types, operators, etc.  Also, views have their own
permissions now, not based on the underlying tables, so permissions on
them have to be set separately.  Check /pgsql/interfaces for some new
ways to talk to PostgreSQL.

This is the first release that really required an explaination for
existing users.  In many ways, this was necessary because the new
release removes many limitations, and the work-arounds people were using
are no longer needed.

Bug Fixes
---------
Fix binary cursors broken by MOVE implementation(Vadim)
Fix for tcl library crash(Jan)
Fix for array handling, from Gerhard Hintermayer
Fix acl error, and remove duplicate pqtrace(Bruce)
Fix psql \e for empty file(Bruce)
Fix for textcat on varchar() fields(Bruce)
Fix for DBT Sendproc (Zeugswetter Andres)
Fix vacuum analyze syntax problem(Bruce)
Fix for international identifiers(Tatsuo)
Fix aggregates on inherited tables(Bruce)
Fix substr() for out-of-bounds data
Fix for select 1=1 or 2=2, select 1=1 and 2=2, and select sum(2+2)(Bruce)
Fix notty output to show status result.  -q option still turns it off(Bruce)
Fix for count(*), aggs with views and multiple tables and sum(3)(Bruce)
Fix cluster(Bruce)
Fix for PQtrace start/stop several times(Bruce)
Fix a variety of locking problems like newer lock waiters getting
	lock before older waiters, and having readlock people not share
	locks if a writer is waiting for a lock, and waiting writers not
	getting priority over waiting readers(Bruce)
Fix crashes in psql when executing queries from external files(James)
Fix problem with multiple order by columns, with the first one having
	NULL values(Jeroen)
Use correct hash table support functions for float8 and int4(Thomas)
Re-enable JOIN= option in CREATE OPERATOR statement (Thomas)
Change precedence for boolean operators to match expected behavior(Thomas)
Generate elog(ERROR) on over-large integer(Bruce)
Allow multiple-argument functions in constraint clauses(Thomas)
Check boolean input literals for 'true','false','yes','no','1','0'
	and throw elog(ERROR) if unrecognized(Thomas)
Major large objects fix
Fix for GROUP BY showing duplicates(Vadim)
Fix for index scans in MergeJion(Vadim)

Enhancements
------------
Subselects with EXISTS, IN, ALL, ANY keywords (Vadim, Bruce, Thomas)
New User Manual(Thomas, others)
Speedup by inlining some frequently-called functions
Real deadlock detection, no more timeouts(Bruce)
Add SQL92 "constants" CURRENT_DATE, CURRENT_TIME, CURRENT_TIMESTAMP, 
	CURRENT_USER(Thomas)
Modify constraint syntax to be SQL92-compliant(Thomas)
Implement SQL92 PRIMARY KEY and UNIQUE clauses using indices(Thomas)
Recognize SQL92 syntax for FOREIGN KEY. Throw elog notice(Thomas)
Allow NOT NULL UNIQUE constraint clause (each allowed separately before)(Thomas)
Allow Postgres-style casting ("::") of non-constants(Thomas)
Add support for SQL3 TRUE and FALSE boolean constants(Thomas)
Support SQL92 syntax for IS TRUE/IS FALSE/IS NOT TRUE/IS NOT FALSE(Thomas)
Allow shorter strings for boolean literals (e.g. "t", "tr", "tru")(Thomas)
Allow SQL92 delimited identifiers(Thomas)
Implement SQL92 binary and hexadecimal string decoding (b'10' and x'1F')(Thomas)
Support SQL92 syntax for type coercion of literal strings
	(e.g. "DATETIME 'now'")(Thomas)
Add conversions for int2, int4, and OID types to and from text(Thomas)
Use shared lock when building indices(Vadim)
Free memory allocated for an user query inside transaction block after
	this query is done, was turned off in <= 6.2.1(Vadim)
New SQL statement CREATE PROCEDURAL LANGUAGE(Jan)
New PostgreSQL Procedural Language (PL) backend interface(Jan)
Rename pg_dump -H option to -h(Bruce)
Add Java support for passwords, European dates(Peter)
Use indices for LIKE and ~, !~ operations(Bruce)
Add hash functions for datetime and timespan(Thomas)
Time Travel removed(Vadim, Bruce)
Add paging for \d and \z, and fix \i(Bruce)
Add Unix domain socket support to backend and to frontend library(Goran)
Implement CREATE DATABASE/WITH LOCATION and initlocation utility(Thomas)
Allow more SQL92 and/or Postgres reserved words as column identifiers(Thomas)
Augment support for SQL92 SET TIME ZONE...(Thomas)
SET/SHOW/RESET TIME ZONE uses TZ backend environment variable(Thomas)
Implement SET keyword = DEFAULT and SET TIME ZONE DEFAULT(Thomas)
Enable SET TIME ZONE using TZ environment variable(Thomas)
Add PGDATESTYLE environment variable to frontend and backend initialization(Thomas)
Add PGTZ, PGCOSTHEAP, PGCOSTINDEX, PGRPLANS, PGGEQO
	frontend library initialization environment variables(Thomas)
Regression tests time zone automatically set with "setenv PGTZ PST8PDT"(Thomas)
Add pg_description table for info on tables, columns, operators, types, and
	aggregates(Bruce)
Increase 16 char limit on system table/index names to 32 characters(Bruce)
Rename system indices(Bruce)
Add 'GERMAN' option to SET DATESTYLE(Thomas)
Define an "ISO-style" timespan output format with "hh:mm:ss" fields(Thomas)
Allow fractional values for delta times (e.g. '2.5 days')(Thomas)
Validate numeric input more carefully for delta times(Thomas)
Implement day of year as possible input to date_part()(Thomas)
Define timespan_finite() and text_timespan() functions(Thomas)
Remove archive stuff(Bruce)
Allow for a pg_password authentication database that is separate from
	the system password file(Todd)
Dump ACLs, GRANT, REVOKE permissions(Matt)
Define text, varchar, and bpchar string length functions(Thomas)
Fix Query handling for inheritance, and cost computations(Bruce)
Implement CREATE TABLE/AS SELECT (alternative to SELECT/INTO)(Thomas)
Allow NOT, IS NULL, IS NOT NULL in constraints(Thomas)
Implement UNIONs for SELECT(Bruce)
Add UNION, GROUP, DISTINCT to INSERT(Bruce)
varchar() stores only necessary bytes on disk(Bruce)
Fix for BLOBs(Peter)
Mega-Patch for JDBC...see README_6.3 for list of changes(Peter)
Remove unused "option" from PQconnectdb()
New LOCK command and lock manual page describing deadlocks(Bruce)
Add new psql \da, \dd, \df, \do, \dS, and \dT commands(Bruce)
Enhance psql \z to show sequences(Bruce)
Show NOT NULL and DEFAULT in psql \d table(Bruce)
New psql .psqlrc file startup(Andrew)
Modify sample startup script in contrib/linux to show syslog(Thomas)
New types for IP and MAC addresses in contrib/ip_and_mac(TomH)
Unix system time conversions with date/time types in contrib/unixdate(Thomas)
Update of contrib stuff(Massimo)
Add Unix socket support to DBD::Pg(Goran)
New python interface (PyGreSQL 2.0)(D'Arcy)
New frontend/backend protocol has a version number, network byte order(Phil)
Security features in pg_hba.conf enhanced and documented, many cleanups(Phil)
CHAR() now faster access than VARCHAR() or TEXT
ecpg embedded SQL preprocessor
Reduce system column overhead(Vadmin)
Remove pg_time table(Vadim)
Add pg_type attribute to identify types that need length (bpchar, varchar)
Add report of offending line when COPY command fails
Allow VIEW permissions to be set separately from the underlying tables. 
	For security, use GRANT/REVOKE on views as appropriate(Jan)
Tables now have no default GRANT SELECT TO PUBLIC.  You must
	explicitly grant such permissions.
Clean up tutorial examples(Darren)

Source Tree Changes
-------------------
Add new html development tools, and flow chart in /tools/backend
Fix for SCO compiles
Stratus computer port "Gillies, Robert" <GilliesR@Nichols.com>
Added support for shlib for BSD44_derived & i386_solaris
Make configure more automated(Brook)
Add script to check regression test results
Break parser functions into smaller files, group together(Bruce)
Rename heap_create to heap_create_and_catalog, rename heap_creatr
	to heap_create()(Bruce)
Sparc/Linux patch for locking(TomS)
Remove PORTNAME and reorganize port-specific stuff(Marc)
Add optimizer README file(Bruce)
Remove some recursion in optimizer and clean up some code there(Bruce)
Fix for NetBSD locking(Henry)
Fix for libptcl make(Tatsuo)
AIX patch(Darren)
Change IS TRUE, IS FALSE, ... to expressions using "=" rather than
	function calls to istrue() or isfalse() to allow optimization(Thomas)
Various fixes NetBSD/Sparc related(TomH)
Alpha linux locking(Travis,Ryan)
Change elog(WARN) to elog(ERROR)(Bruce)
FAQ for FreeBSD(Marc)
Bring in the PostODBC source tree as part of our standard distribution(Marc)
A minor patch for HP/UX 10 vs 9(Stan)
New pg_attribute.atttypmod for type-specific info like varchar length(Bruce)
Unixware patches(Billy)
New i386 'lock' for spin lock asm(Billy)
Support for multiplexed backends is removed
Start an OpenBSD port
Start an AUX port
Start a Cygnus port
Add string functions to regression suite(Thomas)
Expand a few function names formerly truncated to 16 characters(Thomas)
Remove un-needed malloc() calls and replace with palloc()(Bruce)