Out of Memory on two H100 (80GB) each and load_in_8_bit = True
#4
by
Maverick17
- opened
When I run the model in 8 bit, I get an OOM error:
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 106.13 GiB. GPU 0 has a total capacity of 79.10 GiB of which 30.70 GiB is free. Process 1615097 has 2.10 GiB memory in use. Process 132922 has 46.28 GiB memory in use. Of the allocated memory 44.32 GiB is allocated by PyTorch, and 1.38 GiB is reserved by PyTorch but unallocated.