The Secret Sauce

The Secret Sauce

Codd’s secret was relational algebra, a collection of operations that could be used to combine tables. Just as you can combine numbers using the operations of addition, subtraction, multiplication, and division, you can combine tables using operations like selection, projection, union, difference, and join (more precisely, Cartesian join), listed in Table 1-1.

Why did Codd name this relational algebra? Codd based his theory on rigorous mathematical principles and used the esoteric mathematical term relation to denote what is loosely referred to as a table. I’m now ready to define what I mean by a relational database:

A relational database is a database in which: The data is perceived by the user as tables (and nothing but tables) and the operators available to the user for (for example) retrieval are operators that derive “new” tables from “old” ones.1

Examples of Relational Operations

Let’s use the five operations defined in Table 1-1 to answer this question: “Which employees have worked in all accounting positions—that is, those for which the job_id starts with the characters AC?” The current job of each employee is stored in the job_id column of the employees table. Any previous jobs are held in the job_history table. The list of job titles is held in the jobs table.

Given these three tables, you can execute the following steps to answer the business question that has

been posed:

1. This step uses only the employee_id column from the employees table. To do this, you need the projection operation. Following are the SQL command and its results. The employees table contains 107 rows, so this command also produces 107 rows. Only the first five rows are shown here:

select employee_id from employees

100

101

102

103

104

Note that certain formatting aspects of SQL statements, such as lowercase, uppercase, white space, and line feeds, are immaterial except in the case of string literals—that is, strings of characters enclosed within quote marks.

2. This step uses only the job_id column from the jobs table. To obtain this, you need the projection operation again. Following are the SQL command and its results. The jobs table contains 19 rows, so this command also produces 19 rows. Only the first five rows are shown here:

select job_id from jobs

AC_ACCOUNT

AC_MGR

AD_ASST

AD_PRES

AD_VP

3. Remember that the result of any relational operation is always another table. You need a subset of rows from the table created by the projection operation used in step 2. To obtain this, you need the selection operation, as shown in the following SQL command and its results. * is a wildcard that matches all the columns of a table. % is a wildcard that matches any combination of characters. Note that the “table” that is operated on is actually the SQL command from step 2. The result contains only two rows:

select *

from (select job_id from jobs)

where job_id like 'AC%'

AC_ACCOUNT

AC_MGR

You can streamline this SQL command as follows. This version expresses both the projection from step 2 and the selection from step 3 using a unified syntax. Read it carefully, and make sure you understand it:

select job_id from jobs

where job_id like 'AC%'

4. You need the job title of every employee; that is, you need the job_id column from the employees table. The employees table has 107 rows, so the resulting table also has 107 rows; five of them are shown here:

select employee_id, job_id

from employees

100 AD_PRES

101 AD_VP

102 AD_VP

103 IT_PROG

104 IT_PROG

5. Next, you need the employee_id and job_id columns from the job_history table. The jobs table contains 19 rows, so this command also produces 19 rows. Only the first five rows are shown here:

select employee_id, job_id

from job_history

101 AC_ACCOUNT

200 AC_ACCOUNT

101 AC_MGR

200 AD_ASST

102 IT_PROG

6. Remember that the current job of each employee is stored in the job_id column of the employees table. Any previous jobs are held in the job_history table. The complete job history of any employee is therefore the union of the tables created in step 4 and step 5:

select employee_id, job_id

from employees

union

select employee_id, job_id

from job_history

7. You need to join the tables created in step 1 and step 3. The resulting table contains all possible pairings of the 107 emp_id values in the employees table with the two job_id values of interest. There are 214 such pairings, a few of which are shown next:

select *

from

(select employee_id from employees),

(select job_id from jobs where job_id like 'AC%')

100 AC_ACCOUNT

101 AC_ACCOUNT

102 AC_ACCOUNT

103 AC_ACCOUNT

104 AC_ACCOUNT

You can streamline this SQL command as follows. This version expresses the projections from step 1 and step 2, the selection from step 3, as well as the join in the current step using a unified syntax. Read it carefully, and make sure you understand it. This pattern of combining multiple projection, join, and selection operations is the most important SQL pattern, so you should make sure you

understand it. Note that you prefix the table names to the column names. Such prefixes are required whenever there are ambiguities. In this case, there is a job_id column in the employees table in addition to the jobs table:

select employees.employee_id, jobs.job_id

from employees, jobs

where jobs.job_id like 'AC%'

8. From the table created in step 7, you need to subtract the rows in the table created in step 6! To do this, you need the difference operation, the appropriate SQL keyword being minus. The resulting table contains those pairings of employee_id and job_id that are not found in the job_history table. Here is the SQL command you need. The resulting table contains exactly 211 rows, a few of

which are shown:

select employees.employee_id, jobs.job_id

from employees, jobs

where jobs.job_id like 'AC%'

minus

select employee_id, job_id

from job_history

100 AC_ACCOUNT

100 AC_MGR

102 AC_ACCOUNT

102 AC_MGR

103 AC_ACCOUNT

9. Thus far, you’ve obtained pairings of employee_id and job_id that are not found in the employee’s job history—that is, the table constructed in step 6. Any employee who participates in such a pairing is not an employee of interest; that is, any employee who participates in such a pairing isn’t an employee who has worked in all positions for which the job_id starts with the characters AC. The

first column of this table therefore contains the employees in which you’re not interested. You need another projection operation:

select employee_id from

(

select employees.employee_id, jobs.job_id

from employees, jobs

where jobs.job_id like 'AC%'

minus

select employee_id, job_id

from job_history

)

100

100

102

102

103

10. You’ve identified the employees who don’t satisfy your criteria. All you have to

do is to eliminate them from the table created in step 1! Exactly one employee

satisfies your criteria:

select employee_id

from employees

minus

select employee_id from

(

select employees.employee_id, jobs.job_id

from employees, jobs

where jobs.job_id like 'AC%'

minus

(

select employee_id, job_id

from job_history

union

select employee_id, job_id

from job_history

)

)

101

You had to string together 10 operations to produce the final answer: 5 projection operations, 1 selection operation, 1 union operation, and 2 difference operations.

You can express the final answer in a self-documenting way using table expressions, as shown in

Listing 1-1. In practice, you wouldn’t go to such great lengths to restrict yourself to a single relational

operation in each step, because, as you saw in step 7, multiple projection, selection, and join operations can be expressed using a unified syntax. As I said earlier, this approach of combining multiple projection, join, and selection operations is the most important SQL pattern, so make sure you understand it.

Listing 1-1. Final Answer Using Table Expressions

WITH

-- Step 1

all_employees AS

( SELECT employee_id FROM employees

),

-- Step 2

all_jobs AS

( SELECT job_id FROM jobs

),

-- Step 3

selected_jobs AS

( SELECT * FROM all_jobs WHERE job_id LIKE 'AC%'

),

-- Step 4

selected_pairings AS

( SELECT * FROM all_employees, selected_jobs

),

-- Step 5

current_job_titles AS

( SELECT employee_id, job_id FROM employees

),

-- Step 6

previous_job_titles AS

( SELECT employee_id, job_id FROM job_history

),

-- Step 7

complete_job_history AS

( SELECT * FROM current_job_titles

UNION

SELECT * FROM previous_job_titles

),

-- Step 8

nonexistent_pairings AS

( SELECT * FROM selected_pairings

MINUS

SELECT * FROM complete_job_history

),

-- Step 9

undesired_employees AS

( SELECT employee_id FROM nonexistent_pairings

)

-- Step 10

SELECT * FROM all_employees

MINUS

SELECT * FROM undesired_employees

I resume the discussion of SQL in the next chapter. For now, note how formatting improves readability—the formatted version with vertical “rivers” and capitalized “reserved words” shown in

Listing 1-1 was produced using the formatting options in SQL Developer.

What Is a Relational Database?

What Is a Relational Database?

Relational database theory was laid out by Codd in 1970 in a paper titled “A Relational Model for Data for Large Shared Data Banks.” His theory was meant as an alternative to the “programmer as navigator” paradigm that was prevalent in his day.

In pre-relational databases, records were chained together by pointers, as illustrated in Figure 1-6. Each chain has an owner and zero or more members. For example, all the employee records in a department could be chained to the corresponding department record in the departments table. In such a scheme, each employee record points to the next and previous records in the chain as well as to the department record. To list all the employees in a department, you would first navigate to the unique department record (typically using the direct-access technique known as hashing) and then follow the chain of employee records.

Records can participate in multiple chains; for example, all the employee records with the same job

can be chained to the corresponding job record in the jobs table. To list all the employees performing a particular job, you can navigate to the job record and then follow the chain of employee records.

This scheme was invented by Charles Bachman, who received the ACM Turing Award in 1973 for his achievement. In his Turing Award lecture, titled “The Programmer as Navigator,” Bachman enumerated seven ways in which you can navigate through such a database:

1. Records can be retrieved sequentially.

2. A specific record can be retrieved using its physical address if it’s available.

3. A specific record can be retrieved using a unique key. Either a unique index or hash addressing makes this possible.

4. Multiple records can be retrieved using a non-unique key. A non-unique index is necessary.

5. Starting from an owner record, all the records in a chain can be retrieved.

6. Starting from any member record in a chain, the prior or next record in the chain can be retrieved.

7. Starting at any member record in a chain, the owner of the chain can be retrieved.

Bachman noted, “Each of these access methods is interesting in itself, and all are very useful. However, it is the synergistic usage of the entire collection which gives the programmer great and expanded powers to come and go within a large database while accessing only those records of interest in responding to inquiries and updating the database in anticipation of future inquiries.”

In this scheme, you obviously need to know the access paths defined in the database. How else could

you list all the employees in a record or all the employees holding a particular job without retrieving every single employee record?

In 1979, Codd made the startling statement that programmers need not and should not have to be

concerned about the access paths defined in the database. The opening words of the first paper on the

relational model were, “Future users of large data banks must be protected from having to know how the data is organized in the machine (the internal representation)” (“A Relational Model of Data for Large Shared Data Banks”).

Productivity and ease of use were the stated goals of the relational model. In “Normalized Data Base

Structure: A Brief Tutorial” (1971), Codd said,

What is less understandable is the trend toward more and more complexity in the data structures with which application programmers and terminal users directly interact. Surely, in the choice of logical data structures that a system is to support, there is one consideration of absolutely paramount importance—and that is the convenience of the majority of users. … To make formatted data bases readily accessible to users (especially casual users) who have little or no training in programming we must provide the simplest possible data structures and almost natural language. … What could be a simpler, more universally needed, and more universally understood data structure than a table?

As IBM researcher Donald Chamberlin recalled later (The 1995 SQL Reunion: People, Projects, and Politics),

[Codd] gave a seminar and a lot of us went to listen to him. This was as I say a revelation for me because Codd had a bunch of queries that were fairly complicated queries and since I’d been studying CODASYL, I could imagine how those queries would have been represented in CODASYL by programs that were five pages long that would navigate through this labyrinth of pointers and stuff. Codd would sort of write them down as one-liners. These would be queries like, “Find the employees who earn more than their managers.” He just whacked them out and you could sort of read them, and they weren’t complicated at all, and I said, “Wow.” This was kind of a conversion experience for me, that I understood what the relational thing was about after that.”

Donald Chamberlin and fellow IBM researcher Raymond Boyce went on to create the first relational

query language based on Codd’s proposals and described it in a short paper titled “SEQUEL: A Structured English Query Language” (1974). The acronym SEQUEL was later shortened to SQL because—as recounted by Chamberlin in The 1995 SQL Reunion: People, Projects, and Politics—SEQUEL was a trademarked name; this means the correct pronunciation of SQL is “sequel,” not “es-que-el.”

Codd emphasized the productivity benefits of the relational model in his acceptance speech for the

1981 Turing Award:

It is well known that the growth in demands from end users for new applications is outstripping the capability of data processing departments to implement the corresponding application programs. There are two complementary approaches to attacking this problem (and both approaches are needed): one is to put end users into direct touch with the information stored in computers; the other is to increase the productivity of data processing professionals in the development of application programs. It is less well known that a single technology, relational database management, provides a practical foundation to both approaches.

In fact, the ubiquitous data-access language SQL was intended for the use of non-programmers.

As explained by the creators of SQL in their 1974 paper, there is “a large class of users who, while they are not computer specialists, would be willing to learn to interact with a computer in a reasonably high-level, non-procedural query language. Examples of such users are accountants, engineers, architects, and urban planners. [emphasis added] It’s for this class of users that SEQUEL is intended.”

Relational Database Management Systems

Relational Database Management Systems

A First Look at Oracle Database 12c

I want to spend some time talking the theory of relational database management systems. What Leonardo da Vinci said is so important that I’ll quote it again: “Those who are in love with practice without knowledge are like the sailor who gets into a ship without rudder or compass and who never can be certain [where] he is going. Practice must always be founded on sound theory.” How can you competently administer a relational database management system like Oracle if you don’t really know what makes a “relational” database relational or what a database “management” system manages for you?

However, if you’re like most of my students, you won’t be satisfied until you’ve seen an Oracle database management system. I’ll grant that seeing a real system will make it easier for you to understand a few things. But it would take quite a while to coach you through the process of creating an Oracle database management system. Fortunately, there is a solution. Oracle provides a convenient virtual machine (VM) containing a complete and ready-to-use installation of Oracle Database 12c on Linux. All you need to do is to download and install the Oracle VirtualBox virtualization software and then import a ready-to-use VM. The instructions for doing so are at www.oracle.com/technetwork/community/developer-vm. Pick the Database App Development

VM option, and follow the download and installation instructions. (I hope you don’t want me to regurgitate the instructions here.) The instructions are short and couldn’t be simpler, because you don’t need to install and configure Oracle Database; rather, you import a prebuilt VM into Oracle VirtualBox. The only difficulty you may experience is that the prebuilt VM is almost 5GB in size, so you need a reliable and fast Internet connection.

If you follow the instructions and fire up the VM, you see the screen in Figure 1-1. It looks like a

Windows or Mac screen, doesn’t it? This is the Gnome Desktop, which makes it possible to use Linux without getting completely lost in a world of cryptic Linux commands.

Next, minimize the terminal window that’s taking up so much real estate, and click the SQL Developer icon in the top row of icons. SQL Developer is a GUI tool provided by Oracle for database administration. Because this is the first time, it will take a few minutes to start; that’s normal. Figure 1-2 shows what you see at startup.

As shown in Figure 1-3, click Connections, and create a new connection. Give the connection the name “hr” (Human Resources) or any other name you like. Use the following settings:

• Username “hr”

• Password “oracle”

• Connection Type “Basic”

• Role “Default”

• Hostname “localhost”

• Service Name “PDB1”

Expand the Tables item in the navigation pane on the left. Six tables are shown; click the EMPLOYEES table. The data in the EMPLOYEES table is listed in a full-screen editor, as shown in Figure 1-5. If you like, you can make changes to the data and then either save your changes (commit) or discard them (roll back) using the Commit Changes and Rollback Changes buttons or the F11 and F12 keys.

When I was a junior programmer, early in my career, my friends and I were assigned to work on a

big software development project for which we would have to use unfamiliar technologies, although we were promised that training would be provided before the project started. All we knew in advance was that the operating system was something called VAX/VMS; we didn’t know which programming language or database would be used. The very first thing the instructor said was (paraphrasing), “First you have to insert your definitions into the CDD,” and then he walked to the chalkboard and wrote the commands that we needed for the purpose. Needless to say, we were quite flustered, because we had no idea what those “definitions” might be or what a “CDD” was and how it fit into the big picture.

I’m sure you’re eager to learn how to create an Oracle database. Anybody can issue a command such

as CREATE DATABASE or push a button in a GUI tool such as the Database Creation Assistant. But the mere knowledge of a few commands (or even a lot of commands) doesn’t make anyone an Oracle Database administrator in my opinion.

What Is a Database?

Chris Date was the keynote speaker at one of the educational conferences organized by the Northern

California Oracle Users Group (NoCOUG), of whose journal I am the editor. The local television news station sent out a crew to cover the event because Chris Date is a well-known database theoretician and one of the associates of Dr. Edgar Codd, an IBM researcher and the inventor of relational database theory. The news reporter cornered me and asked if I was willing to answer a few questions for the camera. I was flattered, but when the reporter pointed the camera at me and asked, “Why are databases important to society?” all I could think of to say was (paraphrasing), “Well, they’re important because they’re, like, really important, you know.” All those years of database administration under my belt, and I still flunked the final exam!

I’d therefore like to spend a few minutes at the outset considering what the word database signifies. An understanding of the implications of the word and the responsibilities that go along with them will serve you well as a good database administrator.

I’ll begin by saying that databases can contain data that is confidential and must be protected from

prying eyes. Only authorized users should be able to access the data, their privileges must be suitably

restricted, and their actions must be logged. Even if the data in the databases is for public consumption, you still may need to restrict who can update the data, who can delete from it, and who can add to it. Competent security management is therefore part of your job.

Databases can be critical to an organization’s ability to function properly. Organizations such as

banks and e-commerce web sites require their databases to be available around the clock. Competent

availability management is thus an important part of your job. In the event of a disaster such as a flood or fire, the databases may have to be relocated to an alternative location using backups. Competent continuitymanagement is therefore another important element of your job. You also need competent changemanagement to protect a database from unauthorized or badly tested changes, incident management to detect problems and restore service quickly, problem management to provide permanent fixes for known issues, configuration management to document infrastructure components and their dependencies, and release management to bring discipline to the never-ending task of applying patches and upgrades to software and hardware.

I’ll also observe that databases can be very big. The first database I worked with, for the semiconductor manufacturing giant Intel, was less than 100MB in size and had only a few dozen data tables. Today, databases used by enterprise application suites like PeopleSoft, Siebel, and Oracle Applications are tens or hundreds of gigabytes in size and might have 10,000 tables or more. One reason databases are now so large is that advancements in magnetic disk storage technology have made it feasible to efficiently store and retrieve large quantities of nontextual data such as pictures and sound. Databases can grow rapidly, and you need to plan for growth. In addition, database applications may consume huge amounts of computing resources. Capacity management is thus another important element of your job, and you need a capacity plan that accommodates both continuous data growth and increasing needs for computing resources.

When you stop thinking in terms of command-line syntax such as create database and GUI tools

such as the Database Creation Assistant (dbca) and start thinking in terms such as security management, availability management, continuity management, change management, incident management, problem management, configuration management, release management, and capacity management, the business of database administration begins to make coherent sense and you become a more effective database administrator. These terms are part of the standard jargon of the IT Infrastructure Library (ITIL), a suite of best practices used by IT organizations throughout the world.