Overflow and file reorganisation
Introduction
When new files have to be stored in a sequential file, it may not be possible to store them at that moment in time in their correct place. There may not be the space available at the right place in the file. Some files will need to be deleted but can't be deleted straightaway so they are 'marked' to be deleted in the future. Some files are deleted straightaway and leave gaps in the file structure. You may also have the situation where a file is changed, and the change is such that the newly modified file cannot be written back to its original destination. When this situation happens, they are stored in a special area called an 'overflow' area'.
Techniques for managing the overflow area
When a file cannot be stored in the correct place, or a modified file cannot be written back to its original place, the file is saved in the overflow area and a record of where it can be found is placed in the main file area. This can be an ordered overflow area but typically, it is unordered.
Managing the overflow area is an important consideration if file operations are not to quickly slow down and to prevent too many file reorganisations being necessary. When files are created in the first place, they can include some storage redundancy in anticipation of the need to store more files in the future. This is a technique used to reduce the early need for an overflow area. When used, overflow areas are typically split into blocks. One block is filled up and then once full, another overflow block is linked to it and used. Another technique, which is more complex to manage, is that once a block is filled, it is split up into two seperate blocks to create more distributed space. This is called celluar splitting and is more flexible and dynamic than chaining larger blocks together as described before.
The need for file ororganisation
Over time, the overflow area will increase as more files are placed in it. This will become an unsatisfactory situation over time as it will slow down the system; searching through data and sorting it will take longer as the records won't be in their most efficient order for fast searches and sorts. The records will be in two different places, and each of them will need to be looked at. In addition, the gaps created when files have been removed will also slow down file operations and clearly, over time, the number of gaps in the file structure is likely to increase.
Periodically, a sequential file of data will need to be reorganised. This will involve reorganising and writing the file to a new area.
-
- All of the files from both the original area and the overflow area will have to be transfered to the new file and combined into the correct order.
- Any files that have been marked for removal will have to be removed.
- All indexes associated with the data will have to be re-written.
- The old data file will need to be deleted.
- The overflow area will have to be cleared out.