Chapter 10. Assembly of Forces

Coping with Large Volumes of Data

Thenne entryd in to the bataylle Iubance a geaunt and fought and slewe doune ryght and distressyd many of our knyghtes.

Sir Thomas Malory (d.1471)
Le Morte D'Arthur, V, 11

This chapter deals with the particular challenges that are facing us when data volumes swell. Those challenges include searching gigantic tables effectively, but also avoiding the sometimes distressing performance impact of even a moderate volume increase. We'll first look at the impact of data growth and a very large number of rows on SQL queries in the general case. Then we'll examine what happens in the particular environments of data warehousing and decision-support systems.