model.safetensors file's tensors are all odd aligned causing performance issues when mmap'ing the file in place
#18
by
iankronquist
- opened
The tensors in the file model.safetensors have an odd byte alignment in the file. This can cause performance problems when mmap'ing the file in place.
If you look at the first 8 bytes of the file which according to the safetensors spec, represent the length of a json blob header encoded as a little endian 64 bit unsigned integer, you will find that the size of the header is 0x96d5 or 38613 which is an odd number.
Looking at the file near offset 0x96d0 you can see that the bf16 elements of model.embed_tokens.weight begin at byte 0x96dd or 38621.
The solution is to use space characters to pad the size of the json header up to at an absolute minimum to be an even number, ideally maybe even a page boundary.
% shasum model.safetensors
9dd33ff6919a35bb88f9f648b641f809742d1b7f model.safetensors
% xxd model.safetensors | less
00000000: d596 0000 0000 0000 7b22 5f5f 6d65 7461 ........{"__meta
00000010: 6461 7461 5f5f 223a 7b22 666f 726d 6174 data__":{"format
00000020: 223a 2270 7422 7d2c 226d 6f64 656c 2e65 ":"pt"},"model.e
00000030: 6d62 6564 5f74 6f6b 656e 732e 7765 6967 mbed_tokens.weig
00000040: 6874 223a 7b22 6474 7970 6522 3a22 4246 ht":{"dtype":"BF
00000050: 3136 222c 2273 6861 7065 223a 5b31 3531 16","shape":[151
00000060: 3933 362c 3135 3336 5d2c 2264 6174 615f 936,1536],"data_
...
00009670: 2c33 3038 3734 3238 3630 385d 7d2c 226c ,3087428608]},"l
00009680: 6d5f 6865 6164 2e77 6569 6768 7422 3a7b m_head.weight":{
00009690: 2264 7479 7065 223a 2242 4631 3622 2c22 "dtype":"BF16","
000096a0: 7368 6170 6522 3a5b 3135 3139 3336 2c31 shape":[151936,1
000096b0: 3533 365d 2c22 6461 7461 5f6f 6666 7365 536],"data_offse
000096c0: 7473 223a 5b33 3038 3734 3238 3630 382c ts":[3087428608,
000096d0: 3335 3534 3137 3630 3030 5d7d 7d96 3d10 3554176000]}}.=.
000096e0: 3d69 bc9a 3cbd bc9c 3c17 3d8b bd09 bbab =i..<...<.=.....