= new immediate todos: get typing working for wrapper generator demo -> release then in some order: - docs and examples - some basics - roundtrip + report start with users sql: either source files, a database or mixture say if parse ok, -> source files or pg_dumps list type errors from typechecking asts pretty print the catalog from sql source or dumps from typechecking check this catalog against pg catalog also, rename environment type at least - nullability plus support in wrapper generator then maybe: full api review, renames, error handling work, plus sqltypes review = milestones not in any particular order == type check for implicit join: need brain to work on == use better database config command line args and config file for db, username and password, and add prompt style so works like psql == separate out documentation program to separate package -> how to do plugins for extra commands? == documentation work reorganise doc files (changelog, development, questions, README, TODO, usage, introduction, etc.) do test/example doc stuff for parsing and typechecking parse transform pretty print example: basics of approach in chaos - do some notes on how this can be improved hssqlsystem commands reference == tidy files, comment work particularly ag files, redo comments, review all code for crufty bits check todos at top of files: move to bottom == nullability == example code gen for typesafe access == api review tidyups, renames - rename everything pending, review public api function names (particularly environment->catalog, and localidentifierbindings to localenvironment) get rid of inconsistent suffixes in hssqlsystem and command components == error handling work remove all calls to error try to catch all places where errors are being ignored (mainly in the type checking: add these errors to the aast) == roundtrip tests -> report == get chaos stuff working again get chaos sql loading via hssqlppp again so can run roundtrip tests on == module system, show used catalog == better pretty printers for annotation types == extensions code tidy? == annotation generics style == non sql syntax extensions work == catalog ish: triggers, sequences, constraints, index, defaults == haddock work == do makefile type thing for development and deployment of db projects maybe these: builddocs work hssqlsystem commands better cat diff show used catalog finish intro docs haskell intro docs other docs haddock in command components testbattery extensions - tidy code annotation generics: tidy code pretty printer for annotation ppshow less newlines better syntax for extensions modules extension + public/private syntax extension with non sql syntax using preprocessor? multiline strings with string type e.g. sql some sort of report mode to parse, typecheck and roundtrip some sql, and report back what works and what doesn't work on parse errors long term todos milestones for next release todo for chaos (extensions, etc.) divide todo categories investigation, end user docs, internal docs, hygiene/maintenance, testing, new functionality, bug fixes docs: motivation/genesis, history?, high level criteria for 0.1 think about dependencies: make hdbc/pg dependency a flag?, split out exes and tests somehow? = documentation todo write command processor thingy for builddocs add extra commands to hssqlsystem for intro finish intro add haskell intro which uses the commandcomponents instead other documentation files: reference for hssqlsystem hssqlsystem reference is generated from hssqlsystem.lhs: inside this file is the reference information for each command, plus example usage, so these appear next to the actual source. then when building docs, filter this file first to leave only the reference docs, then commandize and pandoc it installation guide do haddock for commandcomponents so it has examples and stuff = New functionality get checkbig mostly working: not trying to get no type errors or differences, but to get all the different aspects of the tests written and running so can see from the output what remains to be fixed main issues: databaseloader isn't working on the chaos sql files anymore, I think this is an issue with the pretty printer producing invalid sql finish extensions: for the tests to mostly work, need to flesh out the current extension implementations which are mainly stubs add missing items to catalog: triggers, sequences, constraints, indexes, defaults = Amending existing functionality add dbms database name, username and password support review holes: round off parsing support for sql - make a list of what needs to be added should try to get as much typechecking working on what parses as possible work on organising the type checking tests, and make the tests much more comprehensive = Housecleaning rename Environment type to Catalog change type annotation from Type to either typecheckfailed Type so can use monad style to propagate typecheckfaileds work on error handling: try to add a lot better error reporting: a few errors are being ignored rather than added to the aast, also try to change all the calls to error into eithers to be added to the aast review commands in hssqlppp: mainly review names and arguments for consistency (i.e. make sure every command accepts first a databasename and then a list of sql files where this makes sense), also redo annotate source so it can be run multiple times on the same source, and run to update the annotation comments when the source is changed. Maybe think about better error reporting? tidy up code, refactor and add lots of comments, particularly in the ag files = Documentation work on haddock documentation find way to do nice html example documentation: short intro/tour/tutorial to give people a flavour of what the hssqlsystem exe can do want to be able to show what the code can do easily and debug annotations - so do a pretty printer which works the following way: output source code on left side of a table (with syntax highlighting, hyperlinks, etc? - use html?) and on the right output the interesting annotations pretty printed, draw lines from the pretty printed annotations to the sourcepositions in the source on the left. Looks like might be able to do this using gtk: http://www.bravegnu.org/gtktext/x498.html - maybe convert to png to allow viewing in a web browser = other issues to maybe fix: check associativity of combine selects and join trefs without parens - write tests using pg_dump output ids in joins issue redo order by parsing/ast to add direction to each element write some notes on hacking: source overview rewrite, file overview rewrites, and rough howto on adding new constructs: altering parser, altering ast and altering type checking process typecheck old and new pseudo records in trigger functions explore creating template syntax for simple expansions of function calls to make writing some extensions easier makefile stuff? plpgsql stuff: blocks, assignments in declarations, strict into, labels,loops,continue look into running test suite from cabal todo for checkbig: get initial catalog tests passing: still some extra table triggers: think it may that foreign keys that pg supports directly and primary keys don't add a table trigger name/text result set resolution issue uniquify constraint names support trim constraint names support distance function return type issue plpgsql language functions support multiple files in annotate source and check source match pgdump output for adding explicit parens e.g. in select unions and table refs, add tests for associativity in the absence of parens tidy up code and docs as usual - lots of work needed on type checking code redo catalog stuff (see below) find some way to produce documentation: want motivation, examples, etc. in nice html get chaos pgdump output parsing and partially type checking, todo: two parsing issues: not not, and count(distinct *) type checking issues exposed: old,new in trigger functions old todoage: ================================================================================ == 2nd next release fix catalog type, lots of work to make more robust: rename environment to catalog better pretty printer for catalog, and for bindings add annotations to catalog, so can see where bits came from do uniqueness testing e.g. disallow two tables with same name, two attributes with same name, etc. fix tests using new catalog approach: don't want to have tests which use sql to alter catalog then use sql to test the catalog changes have been made - use separate tests (more unit-y tests) === types and internal errors handling move type check failed to left of either, make node types an either when there is a type error, add that as separate annotation, and add type as left typecheckfailed. This way can use the haskell type system to check we are chaining type check failed properly. look at all the places where type lists are wrapped in unnamedcomposite and get rid of most of them - just use the type list directly. Look at all the contexts where pg types can be used, and split into separate haskell types, and use sums to put them back together to use the haskell type system better e.g. different set of types can be used for an attribute in a table, for a function parameter, function return type, variable declaration, etc. - want to use the haskell type system to enforce this and get rid of a load of runtime checks (mainly the unwrap setof, etc. in typetype). make sure errors aren't ignored, lots of places where this currently happens, e.g. updating environment - these errors need to end up in the ast. Get all the env functions to return eithers, and stick errors from these in the ast. Convert error calls to either return type and stick these errors in the ast. === parse/typecheck errors to user pretty printers for type check failed, work on error messages from parsing investigate alternative parser libs which can e.g. continue parsing after an error to provide more parse errors at once. write a show for parsec errors which formats the lex tokens and expected lists properly (was broken when moved to the separate lexer) ================================================================================ == other soon todos not sure which ones should be done before 0.1 release. rough guide is 0.1 release should be something that regular sql programmers can try out/use without being overwhelmed with half finished/ missing bits, and actually see something useful. === schemas think of a way to handle schemas, what about changes to schema search path mid code. === catalog and bindings annotation want to add the catalog and current catalog to the annotations, so e.g. we can pretty print what exact updates are made to the catalog from a ddl statement Add the current bindings as annotation everywhere, this can be used by the ambiguous identifier checks, etc., and to understand errors also: add annotations to the catalog so we can see where an entity in the catalog has come from, what update and what source line add similar to bindings, so each binding annotation contains a little explanation for where each binding has come from (e.g. identifier a with type b comes from this tref at source position x,y) === casting annotation for each casting operation, add a log which explains what the relevant algorithm has done to either find the appropriate cast or what it's considered before failing. (e.g. can provide a list of steps taken with the candidate lists in the findcallmatch routine, might be useful for errors when it can't find a match, and if it is calling an unexpected function instead of the one you want). add the prototype of the function actually matched at each funcall site === definition annotations add sourcepos/reference to definition where each item is referenced (table, function, view, identifier binding, etc.) === documentation produce === null checking null treatment Basic motivation is to keep nulls carefully walled off, controlled, and be able to catch them when they sneak back into expressions, etc.. For each value, etc. we determine statically if it might be null. This can be done for return types of functions, fields in a select expression, etc.. (will do mappings e.g. if a functions inputs are all non null, then the output is non null, etc.). Once this is working ok, the second stage is to implement the anti null warnings/ errors. Allow nulls in tables, outer joins, in coalesce, to be produced by selects (maybe add or remove from this allowed list, maybe make it configurable on a per project basis). Never allow nulls to be an argument to a function call, (including ops, keyword ops, etc.). So every time you have a field being used in an expression and it cannot be statically verified to be non null, you have to insert a coalesce or fix it in some other way. So nulls can still be used to represent optional values, n/a, etc.. and output to clients doing selects, but there is no need to grapple with: * 3vl (or whatever it is that sql uses instead), * what the result of a function call is if the some or all the arguments are null, * what the result of a sum aggregate is if some of the values are null, * etc., because none of these things are allowed. === do sql support todo fill in the main gaps, some notes below want to cover '90% of regularly used sql', no attempt for v0.1 to be comprehensive or nearly comprehensive === shadowed identifier test use the bindings annotation to add warnings === modules extension stage 1: set the current module with a function call do creating the module and adding objects to the current module using extension instead of the sql functions used at the moment find a way to annotate the source with modules stuff, want to use it for documentation stage 2: add import/export stuff and check this === other extensions for chaos mark generated code with annotation: make annotation types open with type class or something? - can get rid of typeerrora, statementinfoa wrappers this way check for all ddl, etc. run in functions/executes and turn into extensions so the type checker can see these things out of order definitions: want to write defs in any order and have the extension reorder them into something which can be loaded into postgres finish create_var extension: add constraints constraint system: make a bunch of stuff automatic, like using 'pg accelerators' add notify triggers automatically, including for views convert the sql tests (maybe some of the haskell tests) into static checks on the ast find a better way to work with the tables with null fields: initial values - the copy statements are completely unreadable and uneditable what else? crufty bits to try and simplify: turn sequence progression, action valid tables, ai see if can make the multiple update hack more automatic === development/deployment support: build file for development, just lists sql files in order, and lists which extensions are used deployment: use system to load all sql via extensions into pg, then to a regular pg dump which can then be deployed. === crufty bits to tidy ag files constantly need fixing up for documentation and code clarity syb stuff is a bit messy in astannotations (maybe use uniplate here to reduce dependencies?) tidy the annotating functions in astinternals, implement then in terms of eachother review local bindings use and decide how much logic to implement inside the local bindings module rather than eg. tablerefs.ag want to stop having to write explicit backtrees everywhere. The key to fixing this is some way to read the existing annotation and supply an altered annotation within uuagc code think about how doing multiple passes could help (e.g. star expansion). think about where can use quickcheck review when function bodies are typechecked - see if can do better annotatesource function is a mess, also, make it overwrite it's own annotations instead of adding them so you can run it over a file multiple times. === testing work on making the tests more comprehensive, want a test for each branch/option where feasible === write some automated tests for various roundtrips add script to test chaos sql in various ways: parse roundtrip, load into database via psql and via hssqlppp comparison, and check catalog consistency === do way to produce aast from live database use psql dump, and then can do type checks, lint checks, produce documentation === api review/work on api and do haddock === prepare code for use by non haskell programmers + user docs provide installation instructions do usage documentation file, want to run code embedded in doc source and have results outputted to html output. === prepare code for use by haskell client programmers: user docs as above for using the parser, the extension system, the annotator and annotations, examples for generating sql and writing extensions, etc. === ? place holder statements === precedence and operator parsing add missing operators to parser see if @ as prefix and infix just works in parsec some a_expr and b_expr support properly by looking at pg grammar file --------- some new random notes literal sql mode, with workspace idea: can write code/sql inline in the source, and run it against a database and view the results inline like in some text books where code is executed then the results printed inline into the result - so this is a partial replacement for ghci, emacs haskell-mode ghci, psql, maybe bash, maybe ide, and also a runtime diagnosis tool which can be embedded into production exes. Pretty ambitious, need to focus on useful and easy features. variant: hssqlppp workspace: text view can type in commands and get results back, create multiple workspaces and they're persistent, maybe a wrapper round ghci but not conventional repl style, more like entering elisp and evaluating it in emacs? want to use as replacement for the ghci thing in emacs, and as a shell for working with the ast stuff and actual databases use template haskell to simplify generics in extensions: cases and replacements are a bit verbose and difficult to write and read? = parse and/or type check todo list: (rough/incomplete) "identifier" 6.5e-5 type 'string' style type cast [:] slice missing keyword ops default template1 operators should all parse agg(all expr) agg(distinct expr), agg(*) window frame clauses, named windows parse inside string literals when cast, for common types multidimensional arrays default values serial make sure can type check everything that parses constraint names provide list of keys in info for create/alter table: include unique not null and serials type check fks, and other constraints alter table: add/remove column constraint default value column type rename column rename table what other alters/creates views, functions, operators, types, domains, triggers, rules selects: implicit joins group by, having + group by with unaggregated and aggregated fields distinct, on order by - do properly limit, offset with queries type modifiers data type names with spaces in them timestamps schemas alternative text for true and false enums geometric types weird syntax composites: selector variants, rowctors, component get/update do all keyword and template1 operators any/some/all subqueries and arrays check over rowwise comparisons indexes: create/alter/drop another todo list: catalog: constraints, pg_constraint triggers rules indexes add drops for all creates review which alters to add default columns values + insert checking support pks,fks better support serial properly type modifiers sequences enums listen, notify, pg_listener blocks in plpgsql, assignments in declarations, not null checking strict in into lables, loops, exit, continue raise % counts create schema once all this is done, work on documenting precisely what is supported and what isn't. ================================================================================ example for generating sql code from haskell using the ast get database loader and typesafe access generators good enough to use in chaos example usage of each of these look at the error message formatting, particularly try to fix the parser errors so they make more sense add annotation field to most ast nodes, store type and source positioning in this field, fix parser to add lots of accurate positioning information when parsing. make sure the lint process works on text dumps of databases. try checking the sample databases: http://pgfoundry.org/projects/dbsamples/ ================================================================================ some syntax todo, not organised: ------------ add support for following sql syntax (+ type checking) alter table, common variations create index create rule create trigger + drops for all creates + maybe alters? ctes loop, exit, labels easy ones: transactions, savepoints, listen prepare, execute + using some more: create or replace alter table transactions: begin, checkpoint, commit, end, rollback cursors: declare, open, fetch, move, close, where current of copy - parse properly create database create index create rule create trigger + plpgsql support grant,revoke listen, notify, unlisten prepare, execute savepoint, release savepoint, rollback to savepoint set, reset set constraints set role set transaction correlated subquery attrs plpgsql blocks which aren't at the top level of a function % types strict on intos not null for var defs exception execute using get diagnostics return query execute raise missing bits out params elsif loop exit labels reverse, by in for for in execute expressions: process string escapes, support dollar quoting and other quoting more robustly in the pretty printer full user operator support (?) fix expression parser properly to handle things like between - see grammar in pg source for info on how to do this [:] array slices aggregate: all and distinct multi dimensional arrays: selectors and subscripting missing keyword operators datetime extract time zone subquery operators: any, some, all in general, parsing operators is wrong, the lexer needs to be able to lex sequences of symbols into single/multiple operators correctly, what happens at the moment is a kludge, also, general operator parsing will change how operators are represented in the ast ================================================================================ some other random ideas: more tests: add parse and type check roundtrips automated tests for chaos sql add parse check which parses a single statement, then loads it into pg, pg_dumps it, then parses the output to see if it is the same: since pg sticks plenty of parens in, this should test the associativity and precedence. pg also rewrites some of the sql so will have to match that parser, converter and pretty printer for explain output, want to view how a query is executed in human readable pseudocode. Add lint type checks, etc. to this, which can suggest ways to rewrite the query to get better performance. Another idea is to make the dependencies on the values in the tables more explicit, so you can see how much the data can change before another plan is chosen, or you can see a bad assumption about the kind of data the query will be run on. incorporate pg regression test sql into parsing and type checking tests - can run the roundtrip tests on them, check the documentation generator, etc. redo cabal file to add compile time options: exes, pg support, tests or split into separate packages? work on error reporting, add tests for malformed sql, already some tests and plan for more for sql which fails type checks. want to report multiple parse errors, perhaps can bodge this because of the property that ';' can only appear inside a string or comment, or otherwise at the end of a statement, so add some code to jump to the next end of statement looking ';' and continue to parse to end of file in an attempt to catch at least some further syntax errors get property checker working again - one problem is that the pretty printer will reject some asts (which the parser cannot produce), and the parser will probably reject some invalid sql that the pretty printer will happily produce from some asts. plpgsql on 'roids: write libraries in haskell, and then write syntax extensions for plpgsql using the extension mechanism to access these libs from extended plpgsql e.g. ui lib written in haskell, accessed by syntax extensions in plpgsql then can write the database and ui all in the same source code in the same language, with first class support for properly typed relation valued expressions, avoiding multiple languages and mapping/'impedance mismatch' between database types and types in the language you write the ui in.