Gary Stephenson <gary[AT]stephensong[DOT]com[DOT]au>
ObjectXB is a suite of Xbase++ code that seeks to implement a true RDBMS logical abstraction layer over a set of physical data tables .
It can be downloaded from here (source code distribution only). Note that you will also need to download the ox_runtime.zip runtime files, and the XTX library.
A good demonstration of many of the capabilities of ObjectXB is provided in the GDBU application.
Implemented Features:
The combined effect of all the above features is to make programming with ObjectXB vastly more powerful, efficient and enjoyable than would otherwise be the case. (Of course that's only my opinion, and I'm a wee bit biased! :-)
ObjectXB attempts to provide a complete and rigorous implementation of the "2-Phase Locking Protocol" for transaction management. Each database instance contains its own "transaction processor" which is able to safely interweave its transactions with other database instances (be they threaded or not). (Note that the "physical rollback" functionality is *still* not adequately tested in the Unit Test, as it is seldom actually invoked. Until this situation is rectified, catastrophic failures such as Disk failures and power-outs whilst the database is in the process of actually writing a transaction may still therefore cause data corruption :-( This is high on my list of "round tuits" - a list that seems to continually keep growing despite my best efforts!).
ObjectXB supports class abstractions that I had expected ALASKA to have provided themselves (in their mooted and mourned DBE kit). One day I might write an article about what informs ObjectXB. It would be entitled:
" -> operator considered harmful "
I suspect this to be the main idea motivating for the general push to OODML recently over on the db2k/vDB forums. Whilst the above statement would be true anyway, it is all the more so for the failure of the aforementioned DBE kit to appear.
The use of the "->" operator has thus now been strictly quarantined within the ObjectXB code. Specifically, it is used in the DBFDBE and FOXCDX engines (of course). The FOXCDX driver is also used natively by the transaction tracking and concurrency management implementations, so the "->" still appears within the Transact and dbConnection classes.
ObjectXB currently provides psuedo-DBEs for FOXPRO tables, Clipper NTX/DBT files, and an In-Memory table model. Each table object is bound to a DBE which manages all the driver specific features, allowing Cursors to be opened on table objects without regard to the underlying physical data model being used.
Additional interface to ADS, ODBC and suchlike should be easy to implement, but I just haven't actually got round to doing so as yet. Native C-language interfaces to Oracle and suchlike should be quite feasible, as should also C-code interfaces to C libraries as CodeBase and C-Tree. I also intend to implement interfaces to the Harbour RDDs in the near future
Note that the following are an attempt to describe some guiding objectives for the future evolution of ObjectXB. The current version falls woefully short on many, if not all, counts.
One of the best features of the Xbase++ language is (IMHO) the fact it *omits* the FINALising of methods such as provided in Java. In C++ terms: all Xbase++ methods are implicitly and irrevokably virtual. Thus all classes are fully inheritable (not counting those with "deferred" methods which are not really properly implementable in dynamic languages - they should throw an error at *compile* time, not at *runtime*).
In keeping with this spirit, and taking my lead from truly dynamic languages such as Python and Dylan, I have deliberately eschewed the use of hidden: and protected: features (except perhaps in a few isolated cases where it seemed justified) in favour of exporting the entire interface.
A possible exception here is that I do tend to use READONLY ivars, even though they are something of a misnomer when applied to ivars holding references as opposed to scalar values. The only additional language-implemented encapsulation that I would really like would be to be able to strongly enforce the READONLY ivar concept. This will be impossible for as long as ivars are able to hold Xbase++ object and array references. It is still very useful though IMO to prevent changing the ivars' contained reference (as opposed the object's value or "state") in favour of explicit "setter" methods.
I have also removed as much dependancy as possible from the preprocessor, preferring in almost all cases to implement properly controlled static implementations. I do this owing to some earlier bitter experiences of working with various "pre-processor hacks" in Clipper.
The only exception to the above is that the schema DDL (Data Definition Language) is still implemented as a preprocessor based "little language". As such, it (the preprocessor) has proved to be extremely useful, so we leave it like that for the moment. Recently I have introduced the capability to interpret schema files at runtime - which uses a purpose-built interpreter instead of the preprocessor/compiler combo. For illustration on how to use it, see the included unit test suite.
I have also substantially reduced (in general) the number of parameters accepted by the constructor methods (and indeed all methods) of the component classes, with optional clauses of the DDL declarations being implemented via calls to appropriate "setter" methods etc. This makes the preprocessed code far more readable, and the classes generally far easier to instantiate and configure (i.e. use). Ultimately, I believe the "correct" approach is to prohibit the init method from accepting any parameters whatsoever in favour of specifically name "constructor methods" i.e. class methods that themselves instantiate an instance object, configure it using the supplied parameters, and return it as the result.
In contrast to most other systems I have seen, all assertions - i.e. assert()s, require()s and ensure()s are statically compiled in the source code - and can not be removed/disabled by preprocessor reconfiguration. This is due to the impossibility of guaranteeing side-effect free assertion implementations. (The old story: the code works with the assertions in, but then fails when the assertions are omitted by the preprocessor. Typically the resultant problems will then "pop-up" in candidate releases - the second worst possible place for them to appear.) I believe this approach to be a much sounder solution in most circumstances. Once code is really proven assertions can be commented out by hand as and where required - typically to optimise performance.
I have also eschewed any real attempt to comment the code - hoping and believing that the consistent use of well-chosen descriptive method names with a minimum of input parameters results in code that is very readable and "self-documenting". Some may call this a copout, which I would understand - but I have seen too much wasted effort go in to having to continually maintain comments that really only reflect what the source code is actually doing. Even with diligent attention, the source and the comments invariably seem to drift apart over time, and inaccurate technical documentation is usually worse than no documentation at all - so there!
Readability of code strikes me as being of paramount importance, and is always high on my priorities. It is also one of the main reasons for my never having got much into C or C++ coding.
Comments would be useful in documenting some of the design rationale underlying the particular implementation selected, but in the absence of an automatic documentation tool such as "javadoc", I have opted instead to provide and maintain the document you are currently reading.
IMHO a well commented Unit Test program should be the focus of most (all?) documentation efforts!
I have been studying Date and Darwens' "3rd manifesto" work for quite a few years now, and have high hopes of implementing some really cool Database systems and Data Languages in the not-too-distant future, utilising 3rd Manifesto concepts, with ObjectXB as the code foundation.
One outcome of this study is that I have abandoned my previous attempts to allow Xbase++ objects to form part of the data model as exposed through ObjectXB (using bin2var()/var2bin() as the actual persistence primitives). Instead, the database *must* implement full "expanded object" or "value" semantics, in order to be able to guarantee safe concurrent access capability and predictable scalar behaviours. Conversion of a scalar value to an object (and vice-versa) must therefore happen "outside" the data Model. Object references should *never* be represented as values contained within the database.
This point cannot be over-emphasised - it is one of the most important of the many insights I've gleaned from study of the 3rd Manifesto.
Another subtle but significant message I think is to avoid "side-effect" programming - in favour of a more "functional programming" style. To integrate the concept into our class based object model, it is recommended that any method that *changes* the state of its object, as opposed to simply reporting on it - should return NIL to indicate this fact. In particular the chaining of message sends based on the assumption that the interim chained methods return "self" should be avoided. This is another reason why ivars should always be made readonly - otherwise the "side-effect" of the assignment operator returning the assigned value is brought into play.
The exception to this rule may be in trying to program predicate logic systems, where exclusively returning logical values to indicate success or failure of the called method is mandated by the programming style. This exception is *not* required where we are constructing a "query-only" predicate logic system - which is currently the working model.
Once this "side-effect free" concept is implemented in full (and ObjectXB is currently nowhere near doing so), it is then quite possible to implement sound reliable "constraint" systems, because values obtained by the execution of the constraint logic can be *guaranteed* to be "side-effect" free. This would also then clear the way for the safe and efficient implementation of constraint-based "contracts" (see "Design by Contract").
The very notion of composite scalar values as the sole typed interface to the data model is predicated on a similar notion. By definition, scalar objects cannot "change" their internal state, other than in the sense of "lazily" evaluating internal aspects of themselves. All representations can only "change" a scalar value by generating an entirely new (separate) value.
Other important concepts include "multi-methods", and the "multiple possible representation" abstraction.
As alluded to above, these are (currently) usually implemented via a preprocessor implementation - provided in the #include file "DDL.ch".
A major difference between the current version and previous versions of ObjectXB (and ObjectDB) is in the contents of the DDL.ch. It is virtually a total rewrite. SCHEMA definitions can now be distributed across the various component modules (DLLs), with each module providing its own supporting SCHEMA definition(s). SCHEMA declarations are translated into INIT PROCEDURES, which are merged together into the overall "application schema" at startup time.
There are quite a number classes within ObjectXB, many of which are really "implementation details", and are not part of the user interface per se. Only the most important of the actual interface classes are outlined below. At some stage I might design an external interface for the whole thing (in IDL I suppose), but at the moment there is little or no "packaging" of the classes and features, beyond the class model itself. I also don't currently have any class diagrams available :-(
Basic class that presents and asserts the model (schema), manages connections and threads (independently - see below), oversees the transaction processing model, and provides introspective capabilities to the application for the schema.
In both single and multi-thread modes, the current database object is always to be obtained vir the "db()" function. The actual class of the db() object to be instantiated within the thread can be configured by first calling the setdbClass() function.
The database class also provides (optional) enforcement of maximum concurrent user-limits (licenses) irrespective of whether multi-threaded mode or the (more conventional) multi-user implementations (or both) are being deployed.
Various other features are provided for, including automatic updating of physical table schemas to match those specified by the program, audit logging and failsafe recovery, error reporting and logging, data encryption, exclusive access controls, database duplication, a data replication protocol, backup/restore functionality, and briefcase (offline access) functionality. Note that some of these features are still somewhat "experimental", and are not (yet) tested in the Unit Test program.
In multi-threaded mode, each thread implements a separate connection to the database in the form of an instance of the current database class which incorporates the current thread by multiple inheritance. I experimented with idea of allowing the thread to *contain* a database object, but found the multiple inheritance model more functional and conceptually cleaner. Each thread can be associated with a separate user (remote connections, WTS, Citrix, NT Server, ..)
Multi-threaded programming is very difficult I find, and much attention needs to be paid to detail. I have a separate document I am currently working about the threading model, with an example implementation (from when it was designed).
ObjectXB abstracts out the concept of the physical database driver, and delegates it to a DataBaseEngine object. At the time of writing, the only implemented DBEs are an interface to the ALASKA FOXCDX DBE, and an In-memory Array-based table implementation (see the "ArrayDBE", "ArrayDBO" and "aTable" classes), which are used to support the updatable query concept.
All DBEs ultimately inherit from an abstract "DBEInterface" class. The current implementation provides an interim specialisation of this class called DBFDBE representing all DBF style database engines. This then is further specialised to the (concrete) FOXCDX class.
DBEs contain a "path" property which in the case of DBEs representing phsyical datasets residing in the file system, is the pathto the directory containing the dataset. Table objects refer to their DBE to obtain their file-system path. The database object enforces the requirement that each table have a unique "fullname" (== path + filename), thereby enforcing the concept of each table mapping to a unique dataset. Put another way, no two tables objects are allowed to point to the same - or even intersecting - datasets.
The Unit Test exercises the DBE functionality through the exercising of the Wcontext() class, and elsewhere.
Each separate (logical or physical) table in the system is represented by a single table object. The table object is delegated the responsibility (by the database) for generating contexts (cursors) on itself for particular database objects. Table objects are shared between threads via the database class. In contrast, the contexts generated belong to the thread from which the call is invoked.
Each table object references its own DataBaseEngine object.
Most importantly, table objects maintain of list of component column definitions (i.e the table's Schema), via a contained instance of the "TableSchema" class.
Tables can be configured to use a DBE descended from ArrayDBE(). They can (er.. will) also be provided with a formula (query??) operator enabling them to be populated upon first usage. Thus are they rendered "virtual". There is no actual separate "Virtual Table" class. This may yet prove to be a design flaw, as I have only recently completed the first working implementation.
Provides a raw data "type" definition for reuse amongst columns/tables. Used for introspective, dynamic interfacing to the Model.
This class will ultimately implement a generic "scalar" type system for ObjectXB databases. Initially the scalar type system does not support inheritance, and is object-based (single-tier), not class based. New types are derived by cloning existing element objects, and constraining their behaviour in some way. All derived objects will thus be constrained to conform to a consistent interface. All table columns are required to ultimately derive from (or contain) only scalars.
Instances of this class are required to implement the capactiy to store (and retrieve) a single (arbitrarily complex) "value" of some type. (Think of complex numbers as a good example of the simplest possible "complex scalar"). These *values" may optionally be of variable length - as required by the type.
An important scalar element currently provided by ObjectXB is the "TimeStamp" element, which is required for the (optional) Audit Logging feature, Specialisation of the inbuilt Character and Number elements to particular dimensions can be considered to be a sort of derivation of "built-in" scalar types.
The other builtin types are LONG, ULONG, INT, UINT, DOUBLE, UDOUBLE, DATE, ARRAY and LOGICAL types. (The OxIDE GUI classes also implement a BITMAP scalar type, with a set of defined behaviours - eventually to be published as operators).
Note that ARRAYS are constrained to containing only scalar elements (specifically *not* object references). Whilst they may not seem to really qualify as a Scalar Type, Date and Darwen allow (indeed require) their TUPLE types (a set of name/type/value triplets) to be considered as scalar values. They even extend the definition to include rowsets with an associated Tuple Heading (called "relvars" or, more loosely, "relations" i.e. tables).
The column object essentially binds a contained element to a named column of the table, and provides some optional extras like auto-incrementation, formatters and editors etc.
Autoincrement columns, whilst being a fairly straightforward concept, were particularly hard to implement in a totally correct fashion. This was largely due to the attempt to support the same feature set as found in the original ObjectDB product.
Notwithstanding the seemingly intractable problem of combining support for both off-line "briefcase" modality and autoincrement columns, they also present difficulties in providing generic support for "updatable queries" - the problems being of a similar nature: Autoincrementing within a *copy* of the actual dataset, and then trying to write the data back to the *real* dataset which may itself have issued (conflicting) autoincrement values in the interval between the copy/query and the writeback.
In order to achieve a "provably correct" implementation, I have adopted the following simplifying constraints, which have been arrived at based on my own experiences in implementing and using autoincrement columns a la ObjectDB:
The first of these is not a severe limitation in my experience, I have rarely, if ever, had occasion to have more than one autoincrement field in the same table.
Removing support for the "AUTOINCREMENT BY" construct is more serious, as I have tended to use these all over the place. This because they yield nice consecutive sequences of small values in the tables that just *look right*. The functionality is in fact exactly the same however, the problem being essentially one of presentation which can (I hope) be dealt with straightforwardly in the UI logic.
Given these simplifications, the implementation is as follows:
The system implicitly creates and maintains a table of two columns - "TABLENAME" and "LASTVALUE", containing a record for each table supporting autoincrementation, and the last value issued. This table is explicitly excluded from participation in the Transaction Process logic.
Whenever a new record requiring an autoincrement value is to be created, the system obtains the required value by locating the correct record via the TABLENAME column, and retrieving and incrementing the value in the LASTVALUE column. Note that the actual append operation may still fail for any number of reasons, in which case the issued value will never actually appear in the database. For this reason, it is an error to write code that relies on the sequence of autoincremented values being contiguous.
To support the "updatable query" concept, each table delegates the issuing of autoincrement values to it's linked "writeback" table if it has one. The alias of the first non-writeback table is thus used for obtaining the required value.
Note that this implementation does *not* address the problems of "off-line" briefcase mode, as concurrent, on-line access to the internal AUTOINCREMENT table is required. Briefcase mode may ultimately prove to be unsupportable for this reason. My own need for implementing it has passed, so I doubt that briefcase mode will become available anytime soon.
Probably one of the most useful classes in the library. Provides a powerful facility for active code to be incorporated directly into the Data Model itself (and thus automatically reused across applications referencing that Schema), as Named Attributes. The values are only (and always) calculated upon request. Virtual columns are (usually) calculated at the Server end in Client/Server implementations, with just the actual (scalar) result of the calculation being transmitted across the network link.
Provides for a "caching" of the results of Virtual column calculations. Unlike Virtuals, these columns are (re-)calculated and the result stored as a (read-only) field in the record, whenever the a record is written (:addrec() or :putrec()). This would, of course, only be appropriate when the results of the calculation can be fully-determined exclusively from values contained within the local record i.e other columns. If there are any dependencies on other external data, then the results of the calculation may need to be taken "with a grain of salt". In many cases, however, the performance gain from caching calculated results will justify a certain "lack of timeliness" in the calculated data.
At some stage in the future, it is my intention to remove "indexing" altogether from the exposed ObjectXB interface, as, from a "Relational" perspective they are (should be) an internal "implementation detail" of the Model (owing to our underlying use of an ISAM implementation), and should therefore not be externally visible. For the moment however:
There are basically two concrete index classes supported by ObjectXB - Order() for "real" indexes (persistent - stored on disk) and Vorder() for "virtual" indexes (RAM based only). Both these classes inherit from the abstract Index() class. Virtual indexes can be applied to both virtual and real tables, but Real indexes can only be applied to real tables.
A Virtual index can be deployed on either a "permanent" basis - i.e. associated with a Table instance and accessible to all clients accessing that table, or on a "temporary" basis whereby it will be accessible only from within the particular context instance that created it. The "permanent" usage is reserved for virtual tables only, not real tables.
All Indexes implicitly exclude deleted records. ObjectXB does *not* expose the traditional dbdelete()/dbrecall() functionality to its clients. This is due to the Transaction model's internal dependence on this facility for managing the visibility of newly created records.
Similarly, the Auto-Incrementation functionality also requires internal access to the deleted() record flags. In order to support this, Auto-Increment fields implement their own internal indexes that provide them with privileged access to the deleted records. From a user's perspective, record deletions should be considered permanent as at the time of the transaction being committed.
Indexes support a strong concept of "uniqueness" in that they will explicitly *prevent* updates that would produce duplicate keys. The traditional (weaker) concept of allowing the update to succeed, but omitting the key value from the index can be obtained via the use of the ":ldistinct" ivar (although at the moment this behaviour is not tested by the Unit Test suite).
A Context object implements an (ordered) cursor on a specified table. It uses numerous support classes, and most importantly contains an internal instance of the workarea class, to which it delegates much of its behaviour. A guiding design principle behind ObjectXB is that context()'s should be as "lightweight" as possible, and that the programmer should not be discouraged from creating and destroying new contexts for reasons of performance. ObjectXB recycles disused workarea objects (across threads in multi-threaded implementations) out of a pool of open workareas, thus minimising the use of the (extremely slow) dbUseArea() operation.
Context objects also maintain an internal record buffer, which is automatically loaded from disk as and when required (an automatic "scatter/gather" concept). Contexts are the fundamental component of the transaction machinery.
Contexts can be filtered and/or scoped. Note that record filtering now works quite differently than the traditional xBASE filtering model. All filtering is implemented internally via dynamic deployment of temporary conditional indexes. These indexes are constructed in full at the time that the filter is applied (:setfilter() method), and ObjectXB filtering is thus less dynamic than the normal approach. The advantages are in *much* simpler (read more reliable and maintainable) code, and guaranteed 100% accurate record counts and ordkeyno()s at all times (in contrast to the native reccount()/ordkeyno() functions which ignore both filters and deleted records!) It will depend on the circumstances as to what the effect will be on performance - but mostly I expect it should provide a significant performance improvement over the traditional approach. Given that most knowledgable xbasers tend to avoid filtering (in favour of scoping) wherever possible I believe this approach to be justified.
Contexts can also be inter-related (dynamically scoped to another "parent" context) via the :childof() method. In particular, inter-relating UI table browsers (e.g. "master/detail" tables) is a snap via this technique. Given the "lazy-loading" semantics of the dynamic scoping code used, this technique is also useful when needing to minimize bandwidth requirements between a client and its connected database/Server.
NB - In certain circumstances (e.g. limited file handle environments such as Novell Servers) it may be necessary for the database object to summarily close or reassign a context's workarea without the context being aware of it. For this reason, all references to the workarea object are redirected through an access method which allows for the automatic re-establishment of a suitably configured workarea object before any use is made thereof.
Implements what ALASKA documentation refers to (but does not really expose) as a "DBO" object. Each workarea reference a table object, and delegates all of the requisite data fetching and manipulation operations to the DBE mapped to that table.
This class provides an interface to the In-Memory arrayDBE() architecture. This functionality is fully exercised in the Unit Test to ensure seamless compatibility with the parent context() class.
"Wcontext"s use an internal dynamically created "rowset" object called an aTable. By default each context gets its own separate aTable to play with. "aTables" have writeback capability to a parent cursor of all changes made (via the :update() method).
For a while, ObjectXB included code for a primitive RYO Client/Server implementation - the Server being built around the multi-threaded database implementation, and the Client using a "Proxy" version of the Workarea class to automate it's interactions with the Server. There was also a *very* idiosynchratic RPC implementation, which uses a Socket class, currently interfaced to the "XbTcpIP" library. I expect much will yet change before the code is actually released (if ever).
Whilst the initial test implementation has been very encouraging thus far, I really have no idea how viable or useful this may ulitmately become. IAC I have removed it for the moment so as to not "muddy the waters" more than necessary. There are also unresolved legal issues to do with the use and licensing of the ALASKA Xbase++ runtime :-(.
I am hopeful that the Client/Server implementation may ultimately be the "jewel in the crown" of the whole effort, and am reserving this code for possible future commercial products (bills must be paid somehow!) Only time will tell what happens with this one. If it ever sees the light of day, it will be probably only be available as source-code, and on a subscription basis to the CVS repository. (In the meantime, I would be happy to distribute the code that I have to those who are interested enough to ask for it!)
I did, however, leave in the Multi-Threaded database implementation on which the Server is actually built, because it is so useful for automating concurrency and load testing etc. Whilst it does *not* completely obviate the need for manual testing, it certainly provides a much safer and easier environment within which to experiment and develop good test cases. I expect it might also make it far easier to plumb applications to other DBMS's etc. (I had a good example of how to make best use of the multi-threaded database capability, which I will try to dig out and publish as a separate read-me asap.)
A (somewhat rudimentary at the moment) test program is provided to exercise the ObjectXB module. Ideally, as we encounter problems, we first reproduce them in the test program. Only then is the source code modified to pass the newly modified Unit Test. Unforunately, following this ideal process has not always been feasible - but the more we do it, the better we get at it.
This code has some history to it, being based on the prior ObjectDB Clipper code (the product was first released Circa 1992, with the source code becoming available a couple of years later).
I released the first version of my Xbase++ implementation of it,to a deafening silence ;-) several years ago now. In retrospect, it was a fairly naive and amateurish attempt. I have continued to work on it over the intervening period, and whilst I am sure it remains naive and amateurish in may ways, it has been dramatically improved. It is now a *much* more mature and capable piece of software. Having said that, I fully expect the silence this time to be even more deafening, but this is not really of concern to me, as my ultimate goal is to use ObjectXB as a basis for other projects.
I am releasing the code simply to honour certain commitments I made in the past, both to myself and others - most particularly to Brian Marasca (thanks again Brian!). Unfortunately, ObjectXB still faces much the same problem as the original ObjectDB - it is basically too difficult to port existing (non OOP) code to it, and there is litle or no supporting introductory material. So be it!