npak
It is odd I talk nuts and bolts here but here we go.
I have been playing with the idea of a backup file format that can store multiple files from multiple back ups. The biggest up side of this is space; Why backup redundant data? Create some low-overhead method of detecting changes, do the work once, and store it. The biggest down side is that there is no fantastic way to do this without either a ton of disk reads/writes or tons of memory. Since most users do not have 4GBs of RAM but do have a lot of free disk space the answer seems obvious. Of course disk writes are much, much slower than primary storage but there is no other portable solution for the average PC.
Here is the pre-pre-release file format I am currently kicking around:
================================= Terminology =
================================
Page:
Collection of files within a pack-style format.
Chunk:
Files are broken down to chunks of N size and hashed. These hashes are stored and used to determine if a chunk of a file has changed.
================================
= File Format =
================================
String: "npak"
Integer: [Major Version]
Integer: [Minor Version]
Integer: [Chunk Size]
Integer: [Number of Pages]
foreach ( page ) {
Integer: [Page Size]
[Page]
}
String: "end npak"
================================
= Page =
================================
String: "start page"
foreach ( file ) {
Integer: [File Name Length]
String: [File Name]
Integer: [Date Created]
Integer: [Date Modified]
Integer: [Date Accessed]
Integer: [File Location]
Integer: [File Length]
foreach ( chunk ) {
Integer: [Chunk Hash]
}
}
[File Data]
String: "end page"