node-haystack Episode 5: Volume

This article talks about the structure of volume file.
Layout
Volume is the entity for service. A volume includes a lot of blocks. Blocks store the exact data.
Volume
The layout of a volume file looks like this:
Field
Size
Description
Magic number of volume
8 bytes
0x6b63617453796148 or { ‘H’, ‘a’, ‘y’, ‘S’, ‘t’, ‘a’, ‘c’, ‘k’ }
Block 0
Varies
block 1
Varies
…
Varies
block N
Varies
Block
The layout of a block looks this:
Field
Size
Description
Magic number of header
4 bytes
0x53796148 or { ‘H’, ‘a’, ‘y’, ‘S’ }
Key
16 bytes
A uuid value
Cookie
4 bytes
Cookie for avoiding attack
Tag
2 bytes
Optional field for user
Flag
2 bytes
If removed
Data size
4 bytes
Length of data, in byte
Check sum
4 bytes
Parity check sum
Data
Varies
The content
Padding
Varies
0 - 7 bytes, make sure the blocks aligning at 8 bytes boundary
Magic number of footer
4 bytes
0x6b636174 or { ‘t’, ‘a’, ‘c’, ‘k’ }
NOTE The data size field of block is a 32bits integer. That means the maximum size of data field of block is 2^32, or 4GB.
Index
To speed the finding of blocks, a hash is used to index the blocks. The key type of hash is std::string . The following information of a block will be kept in hash:
Field
Size
Description
Key
16bytes
The uuid key
Cookie
4bytes
Tag
2bytes
Flag
2bytes
Since the memory will align at 32bits boundary at least, this field should be harmless. 0x0000 for normal, 0x0001 on removed.
Size
4bytes
Position
8bytes
The offset of beginning of the block in volume file.
Cache
Implementing a C++ cache is painful. The flexible high-level language JavaScript will be a better choice, even it may cause the lost of performance a little bit. The cost is worth.