Sigmoid Loss for Language Image Pre-Training (original) (raw)
Abstract:We propose a simple pairwise Sigmoid loss for Language-Image Pre-training (SigLIP). Unlike standard contrastive learning with softmax normalization, the sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization. The sigmoid loss simultaneously allows further scaling up the batch size, while also performing better at smaller batch sizes. Combined with Locked-image Tuning, with only four TPUv4 chips, we train a SigLiT model that achieves 84.5% ImageNet zero-shot accuracy in two days. The disentanglement of the batch size from the loss further allows us to study the impact of examples vs pairs and negative to positive ratio. Finally, we push the batch size to the extreme, up to one million, and find that the benefits of growing batch size quickly diminish, with a more reasonable batch size of 32k being sufficient. We release our models at this https URL and hope our research motivates further explorations in improving the quality and efficiency of language-image pre-training.
Submission history
From: Xiaohua Zhai [view email]
[v1] Mon, 27 Mar 2023 15:53:01 UTC (312 KB)
[v2] Thu, 30 Mar 2023 17:03:49 UTC (332 KB)
[v3] Thu, 4 May 2023 17:39:26 UTC (324 KB)
[v4] Wed, 27 Sep 2023 12:05:41 UTC (601 KB)