Historical Perspective and Modern Developments

Databases were originally collections of "flat files", what we would think of as a spreadsheet, with columns (fields) and rows (records/data). The flat files were manipulated by programs that were typically written in the programming language COBOL.  Typically, multiple copies of the same data were maintained, each copy sorted in a different way. The invention of the index (a data structure on disk that enabled one to look at the data as though it were sorted on one or more fields) changed all that. A single flat file could have as many indexes (on as many fields) as necessary, making multiple copies unnecessary.

Hierarchical Databases (mid 60s)

The first attempt at a database was IBMs IMS, which followed what was termed the hierarchical model. This allowed one-to-many relationships (e.g., one manager, many subordinates) but had a problem with many-to-many relationships. (E.g., if a subordinate reported to more than one manager, you had to duplicate the subordinate's data.)

Having said this, there are certain kinds of data that naturally follow a hierarchical model, notably, documents that are marked up with XML. There are times when you want to manipulate the XML (e.g., performing queries on a single document to extract certain features), and you often want to index the XML content for faster search across documents that are stored in the same database. Most high-end relational databases (see below) now offer XML support to a varying degree.

Network Databases (late 60s)

The Network model, developed in response to this shortcoming, allowed many-to-many relationships. They are fast and efficient (network databases are still used to this day in applications such as satellite communications and airline reservations, where very fast response times are needed). This is partly because they use "hard pointers": references to the contents of one table from another actually refer to the physical address of the relevant data on disk, bypassing the need for indexes.

However, network databases require a lot of programming to use successfully- unlike relational databases, they cannot be used by non-programming power users out of the box. Also, they are inflexible: once the data is organized in a particular way, it is hard to change one's mind. Data reorganization involves the equivalent of major surgery.

Relational Databases (late 60s to present)

Today's de facto standard databases were conceived in late 60s by Edgar F. Codd at IBM. Codd was a mathematician, and the heavily mathematical tone of his papers was partly responsible for the somewhat delayed acceptance of his work. Codd's idea was to define a set of operations that would work on databases the way algebraic operators worked on numbers. The first pilot implementation (at IBM San Jose by a team led by Chris Date in the early 70s) demonstrated the viability of the concept. Date was to subsequently play a very important role in popularizing Codd's ideas and translating them into accessible language with examples that showed off the ideas'  elegance.

Codd's original papers used mathematical (Greek or Greek-inspired) symbols as database operators. Apart from the fact that programmers weren't mathematicians, there was also the practical consideration that such symbols couldn't be readily entered via computer keyboards, and so it was decided to create something more accessible, and COBOL provided an obvious inspiration. COBOL, created in the 1950s, was a programming language that was intended to be English-like, the idea being that ordinary people, for example, businessmen, could learn how to program if the constructs used to create a program could be expressed in something similar to sentences in ordinary language. (This idea turned out to be simplistic: it was soon discovered that expressing complex ideas in any programming language ultimately requires a level of involvement and expertise that has relatively little to do with the choice of words used for expression, as long as the choice of those words is not completely counter-intuitive. In other words, once a programming task becomes elaborate enough, amateurs are unable to perform it. COBOL's design only achieved verbosity without appreciably simplifying the challenge of programming.) 

In keeping with the ideas of COBOL, however, relational database implementation was thought to be facilitated by creation of an English-like Query Language. The language created for this purpose was called SQL (Structured Query Language), though it eventually grew in scope to handle other tasks such as editing/modification of data, data definition (e.g., specifying the structure of individual tables) and database security (e.g., specifying the privileges of individual database users for different tables in the database). SQL is reasonably well-designed, however: at the level of data manipulation, it has a limited number of words and concepts that can be learned quickly, which is a tribute to the power of Codd's ideas. More important, because the essential components of the language are so few, they translate quite well to a graphical metaphor: that is, a database, or a subset of it, can be represented as a diagram, and software tools can allow a user to compose SQL through a process that is mostly point, click and drag.

Object-Oriented Databases (OODBs) and Object-Relational Databases (late 80s to present)

Object-orientation is a programmer's rather than an end-user's concept. To understand this term (which has been one of the buzzwords in database marketing during the 90's), you must first understand the concept of a class. An class is a "kind of thing" that we want to represent information about- a student, employee, product, etc. To use an over-simplification, it is roughly the equivalent of a table in a relational database, with an important enhancement: the permissible operations on a class ("methods") are also specified along with the class data structure definition. That is, class = data structure + methods. An object is an instance of a class, e.g., Jim Jones is an instance of Employee.

An Overview of Classes

From the programmer's point of view, classes are a good thing. This is because a well-designed class is reusable: instead of reinventing the wheel, you use other people's work. More important, a properly designed class  hides the details of how it works, and lets the user of the class focus on the more important task of getting the job done. All that the user of the class has to know is what the methods of the class are good for, and how to use them. An example of a well-designed set of classes is the set that is bundled with Microsoft Word. While most people use Word only to compose documents, it is also possible to program Word to do pretty interesting and powerful things that would take you a very long time to create if you started from scratch.

 For example, there is a method in Word called GetSpellingSuggestions, which takes a series of letters representing a word, and can do one of several things: 

The interesting thing about this method is that it takes just a single line of program code to call it: a tremendous amount of work is being done behind the scenes, but the programmer doesn't have to worry about what's happening: only the end-result matters.

On platforms such as Windows, classes can be designed in such a way that they can be called in a programming language other than that in which they were originally written. For example, though Word is written in a combination of C and assembler, you can call  GetSpellingSuggestions from Word's own programming language, Visual Basic for Applications. (For that matter, you could call it from a custom application that you built: such an application could be used as part of a Medical Dictation System, for example, to increase the accuracy of transcription.)

Classes and Databases

Relational databases have traditionally addressed only the data structure aspect of classes. For large and complex systems, bundling the operations along with the data structure enables more robust construction of the system components (because each component can be tested more thoroughly before it is added to the system). Almost all mainstream programming languages (even Visual BASIC) have evolved to incorporate object-oriented features, and OODBs have similarly evolved to fill a lacuna.

Unfortunately, many of the early "object-oriented" DBs were little more than repackaged hierarchical or network database engines with a modest programming veneer layered on. Mercifully, market pressures have resulted in considerable attrition. Also, most relational databases (Oracle, Informix) have evolved to add object-oriented features, resulting in the Object-Relational database (ORDB).

OODBs and ORDBs are a heterogeneous bunch, and the ones that have survived have done so for several reasons:

1. The ability to model and manipulate complex data, such as images, video, fingerprints and so on. The main advantage of OODBs here is that allow considerably greater control over the storage and manipulation of the contents of a complex data element than most relational systems would. For example, one can think of "methods" for a fingerprint: returning a classification code for a fingerprint of a known or unknown person (so that it may be filed away), and searching the database with the fingerprint of an unknown person to return close matches, which can then be checked by human experts.

2. Support for special operations common to niche markets that are not very well supported by the more mainstream database engines, e.g., support for Computer-Aided-Design and Manufacturing (CAD/CAM) and Document management. Both of these involve Version Control, a concept that involves tracking multiple versions of something, being able to report the differences between two versions, and so on.

3. Support for data structures such as arrays, which are useful for waveform data such as EKGs and EEGs.

4. The OODBs with an underlying network architecture use "hard pointers" for speed of retrieval in operations such as telecommunications. A hard pointer is something close to an actual address on a disk drive, so that the software can more or less go directly to the data rather than having to first compute its location through an indirect process.

Object-Relational Databases

Originally, relational databases did not let you do anything with image or sound data other than store them and fetch them. As far as the database was concerned, it did not know, or care, that the data was a wave signal, an image or a sound: it just treated it as a bunch of bytes of arbitrary size. The term "binary large object" or BLOB, was coined to describe database fields that stored such data. Manipulation of the data was left to special programs on your machine: all that the database did was hand the data over to them. The developer could not extend the database vendor's dialect of SQL to allow operations on such data, (e.g., "show me all records that have the same fingerprint classification as the current fingerprint").

This changed with the appearance of the program Illustra (itself a commercialization of the University of California, Berkeley, POSTGRES project). Illustra was subsequently bought by Informix and incorporated into the Informix Universal Server technology (and later, Informix itself was acquired by IBM). The essential idea (exemplified by Illustra's marketing logo, "No BLOBs") is that there should be no difference (with respect to the level of difficulty) between the manipulation of simple datatypes and complex datatypes.

Of course, not everyone has the skills and gumption to program the manipulation of video or sound clips. Therefore, the idea was to allow hardcore developers to create "database extensions". An extension is basically a programming toolkit that had all the functionality for manipulation of a special kind of data (e.g., multimedia, spatial data). Such extensions can then be (and are) made commercially available for less skilled programmers to use.

Oracle, for example, markets extensions for handling spatial data as well as free text data such as bibliographic data. Informix also markets extensions which can handle text in a variety of native word-processing and other formats (e.g., HTML, MS-Word, Wordperfect).

It is well known that nobody gets a program right the first time, and extensions are basically very advanced programs. With Illustra, creating an extension was something akin to pulling off a heart-lung-liver transplant; one little mistake and your database came crashing down, so that the development paradigm consisted mostly of writing code, praying that it would work, and restarting your machine and database when it didn't. With Oracle, things weren't much better, though you could create such extensions in the Java programming language, which is a lot "safer" (i.e., less crash-prone) than the language C, which Illustra mandated.

The fact is that, if a developer decides to write extension, s/he needs all the help one can get. An integrated development and debugging environment does a lot to improve productivity, and when Microsoft got in on the object-relational bandwagon with SQL Server 2005 and later, it provided such an integrated environment through Visual Studio .NET. The .NET languages, C# ("See-Sharp") and Visual Basic.NET follow the Java design of being inherently safe, so that the programmer still has a safety net.

In any case, most of today's mainstream relational databases are now object-relational.