fluidb-old/snapshot/DESIGN.md

Snapshot()

If the Snapshot Store (SS) needs a full snapshot, then:

• checkpoint the WAL
• set the path to the SQLite file in the FSMSnapshot object

If the SS needs an incremental snapshot

• If WAL is larger than SQLite file: (OR compact WAL -- will be the minimum changes required to copy)
• copy WAL bytes to snapshot (using memory for storage)
• checkpoint WAL into SQLite file.

Persist()

• create .tmp directory in SS (already created by Sink Open())
• check type of snapshot
• if full:
• move SQLite file to SS root, named as .db
• delete .db, if it exists, and any earlier snap directories too.
• else:
• move WAL file to beside existing SQLite file. Name it .db-wal. Sync everything.
• rename .tmp to
• rename .db to .db
• checkpoint .db-wal .db
• open and close .db to remove any WAL file
• delete any older snapshot directories <snap-1....-n>

Recover()

• if there is a directory ending with .tmp:
• call it
• indicates an incomplete snapshotting process
• delete all files and directories that contain the string
• get the newest snapshot, if any
• call this
• there should be a DB file named .db. If not rename .db to .db. If there is no .db file, panic?
• if there is a WAL file called .db-wal checkpoint it into .db.
• open and close .db to remove any WAL file
• delete any older snapshot directories <snap-1....-n>, including any <snap-1......-n>.db

Misc:

• need to lock between Open()/Close() and a Sink existing, until Persist/Release is called.

COMPACTION:

Let's discuss the SQLite WAL file. When I say "compact the WAL file", what do you think I mean? ChatGPT

When you refer to "compacting the SQLite WAL (Write-Ahead Logging) file," I interpret this as the process of reducing the size of the existing WAL file by eliminating redundant or unnecessary frames. Specifically, if multiple frames in the WAL file pertain to the same database page, these could be reduced to a single frame that contains the most recent state of that page. This operation would result in a new WAL file that is more space-efficient while still ensuring that all changes represented in the original WAL are retained. This is in line with the typical purpose of file compaction in various storage systems, which is to improve space utilization and potentially read/write performance. User Give me a step by step process for compacting a WAL file. ChatGPT

Certainly. Here is a step-by-step procedure to compact a SQLite Write-Ahead Log (WAL) file:

Initialize: Create a new, empty WAL file to store the compacted frames.

Read Header: Read the header of the existing WAL file to identify its format, version, and other essential metadata. Copy this header information to the new WAL file.

Create Frame Map: Initialize an empty map to keep track of the most recent frame for each page number in the existing WAL file. The map's keys will be the page numbers, and the values will be the corresponding most recent frames.

Iterate Through Existing WAL: Read the existing WAL file frame by frame. For each frame:
    Extract the page number from the frame header.
    Check the frame map to see if this page number already exists.
    If it does, replace the existing frame in the map with the current frame.
    If it doesn't, add the current frame to the map.

Write to New WAL: Iterate through the frame map and write each frame to the new WAL file. This should be done in the same order the frames appear in the original WAL file to maintain any implied dependencies between the frames.

Update Checksums: Compute and update the checksum for each frame in the new WAL file. This is crucial for maintaining the integrity of the data.

Finalize New WAL: Once all frames have been written and checksums updated, close the new WAL file.

Validation: Optionally, you may wish to validate the new, compacted WAL file by applying it to a test database and ensuring it brings the database to the expected state.

Swap Files: Once you're sure the new WAL file is accurate, you can replace the existing WAL file with the new one. This step should be done with caution, considering the implications on database consistency.

Checkpoint: After the swap, perform a SQLite checkpoint operation to ensure that the changes from the new WAL are applied to the database.

By following these steps, you should have a compacted WAL file that maintains the integrity and semantics of the original one. User Let's talk about step 5. Imagine I have a WAL file with these pages. The digit in the page is the page number, the letter is the version. Version, say, b, is newer than version a. Here is the sequence: 1a,1b,2a,4a,5a,1,c,6,7,6a,8,9 List the pages, in sequence, as they will appear in the new WAL. ChatGPT

In the compacted WAL, the goal is to retain only the most recent frame for each page. The sequence in the existing WAL is as follows:

1a, 1b, 2a, 4a, 5a, 1c, 6, 7, 6a, 8, 9

For page 1, the most recent frame is "1c". For page 2, the most recent frame is "2a". For page 4, the most recent frame is "4a". For page 5, the most recent frame is "5a". For page 6, the most recent frame is "6a". For page 7, the most recent frame is "7". For page 8, the most recent frame is "8". For page 9, the most recent frame is "9".

In the new, compacted WAL, the sequence of frames would be:

1c, 2a, 4a, 5a, 6a, 7, 8, 9

It should be noted that the order in which these frames appear in the new WAL depends on the sequence in the original WAL, which is important for maintaining any implied dependencies between the frames. User Page 2a, for example, comes after page 1a and 1b in the original WAL file. However it will come after page 1c in the rewritten WAL file. Will this be an issue? ChatGPT

The sequence of frames within a WAL is significant because the purpose of the WAL is to record changes to a database in the order they occurred. This ensures that the database can be correctly reconstructed from a snapshot and the subsequent changes logged in the WAL. If the sequence of frames is altered during the compaction of the WAL, it may disrupt this order, potentially causing issues when SQLite attempts to replay the WAL to bring the database to a consistent state.

Therefore, if a compacted WAL is to be used in place of the original, it is critical to maintain the original sequence of the most recent frames for each page. In your example, the sequence "1c, 2a" in the compacted WAL correctly reflects that changes to page 1 should be applied before changes to page 2, just as "1a, 1b, 2a" did in the original WAL. This ensures that dependencies between frames are preserved.

Thus, while compacting a WAL can remove redundant frames, altering the sequence of remaining frames is generally not advisable due to the potential for disrupting the order in which changes should be applied. User Right, but before changes are applied to page 2, those changes will also include changes from page 1c. Is this going to be an issue? ChatGPT

In a typical Write-Ahead Logging (WAL) implementation, each frame represents a change made to a particular page at a specific point in time. The frames are applied in sequence to ensure that the database reaches a consistent state that reflects all changes made.

If frame "1c" is more recent than "2a" and both frames are applied during the same WAL replay, then "1c" should be applied before "2a" to achieve the most recent and consistent database state. This is in line with the original sequence of changes as they were made to the database.

If, however, there are dependencies between pages—such as constraints, triggers, or transactional consistency requirements—that require changes to page 1 to be applied in a certain relation to changes to page 2, then the sequence could matter. In such cases, disrupting the sequence could result in a database state that violates these dependencies.

For the SQLite WAL specifically, SQLite should be able to properly apply the changes as long as the WAL frames are sequentially correct for each page. SQLite performs atomic commits by applying all changes in a WAL file as a single transaction during a checkpoint. Therefore, as long as the integrity of individual transactions is maintained within the compacted WAL, applying frame "1c" before "2a" should not be an issue.

In summary, it will not be an issue if frame "1c" is applied before frame "2a" as long as there are no explicit dependencies between these pages that rely on a different order of operations. User So from your step-by-step instructions above, it sounds like I also need, say, as slice to keep track of the order of the pages I read from the WAL. Keys in a map do no guarantee any order. ChatGPT

Indeed, the order of keys in a Go map is not guaranteed to be consistent. Therefore, using a slice to keep track of the order of pages as you read them from the original WAL is advisable. This ensures that, when you write frames to the new, compacted WAL, you maintain the original sequence of page changes.

In summary, a slice can serve as an ordered log of page numbers you encounter, ensuring that the frames in the compacted WAL file are written in an order consistent with their original sequence. This is particularly important for preserving the integrity of transactions and any other dependencies between the pages. User Is there any other way you would suggest? Any other Go type? ChatGPT

Another Go type to consider for maintaining the order of pages is a slice of structs, where each struct contains the page number and associated metadata, such as the version or offset in the original WAL. This would allow you to keep additional contextual information alongside the sequence, which might be beneficial depending on your specific requirements.

For instance, you might define the struct as follows:

go

type PageMeta struct { PageNumber int Version string // any other metadata fields }

You could then use a slice of these structs to maintain order and metadata:

go

var orderedPages []PageMeta

This approach allows you to both maintain the order of pages and keep any additional information that might be important for the compaction process or for debugging.

It's a slight expansion upon the basic slice of integers approach but offers more flexibility.