Preferred dtype: bfloat16 or float16?
#72
by mihaidusmanu - opened
What's the preferred dtype for this model? Readme mentions bfloat16 in the table but the transformers demo uses float16 at model initialization.
mihaidusmanu changed discussion title from Preferred dtype bfloat16 or float16 to Preferred dtype: bfloat16 or float16?
I tested both on some small test datasets and interestingly they seemed to be very similar in performance, but would still be interested to have an answer!
Hi @mihaidusmanu , yes both bf16 and fp16 perform similarly. fp16 is more widely supported, which is why we suggest it in the demo. It also provides higher precision and greater consistency across different configurations (hardware, batch size, etc). The main weights are given in bf16, as this was used during training and is preferable for quantization or further fine-tuning.