The Limits of Cross-Lingual Transfer: Evaluating SignCLIP on LIS

Introduction

Sign Languages are the primary means of communication for millions of deaf individuals worldwide [1], [2]. Isolated Sign Language Recognition (ISLR) remains an open research area at the intersection of computer vision (CV) and natural language processing (NLP) [1], [3]. Similar to spoken language research, there is a large discrepancy in the efficacy of state-of-the-art solutions between high-resource and low-resource languages. Unlike the widely studied American (ASL), British (BSL) and Chinese Sign Language (CSL), Italian Sign Language (LIS) remains under-resourced, lacking a large-scale, annotated corpora required for the training of a deep neural network that can recognise it effectively [2].

To overcome this limitation, recent research has pivoted toward transfer learning and few-shot recognition, leveraging models pre-trained on large multilingual datasets [4]. One of these models is SignCLIP, which utilises contrastive learning to project spoken language text and sign language videos into a shared embedding space. It is pre-trained on Spreadthesign, a dataset containing approximately 500,000 video clips in up to 44 different sign languages [1], including LIS. However, downstream evaluations for LIS recognition and text-video retrieval tasks were entirely omitted in its original benchmarks [1].

Consequently, the applicability of these multi-lingual priors to low-resource LIS datasets remains unexamined. To address this research gap, we present the first investigation for LIS that explores the performance of zero-shot, few-shot, and fine-tuning paradigms using SignCLIP as a foundation model, evaluating both cross-modal Video-Text retrieval and ISLR. For this evaluation, we used two datasets: A3LIS-147, introduced in [5], and SignIT [2]. These datasets enable complementary evaluations by contrasting a controlled, balanced multi-signer environment with domain-specific vocabulary against naturalistic, unbalanced, core-vocabulary signs, respectively.

Our main findings are as follows:

• Empirical benchmarking of SignCLIP for LIS: Our results suggest that aligning diverse multilingual signs to a monolingual text space induces negative transfer. While cross-lingual iconicity aids specific vocabulary subsets, performance stratification indicates that cross-lingual semantic divergence in sign motion primitives acts as a bottleneck during training. Nonetheless, highly competitive few-shot results confirm that the backbone successfully preserves discriminative spatial embeddings.
• Ablation of fine-tuning strategies: Through extensive optimisation ablations, we demonstrate that a Global Noise-Contrastive Estimation (GlobalNCE) objective yields superior adaptation for low-resource sign languages. By maximising hard-negative density within the global contrastive space, GlobalNCE most effectively organises the embedding space.

Related works

Italian sign language recognition

LIS ISLR research has largely targeted small-scale, controlled settings, utilising the A3LIS-147 dataset as the primary benchmark [6], [7]. Early approaches with Hidden Markov Models (HMMs) have been improved upon by more recent work reaching an accuracy of 80.4% with fully-supervised CNN models (Inception3D and SlowFast) [6].

The above work suffers from several structural limitations in the context of scalable SLR. The architectures employed are incapable of adapting to out-of-dictionary vocabulary without retraining. Furthermore, they are optimised for clean artifact-free datasets, potentially suffering in performance with out-of-distribution noisy data seen during real-world deployment [8].

To address the latter, the SignIT dataset was recently introduced to benchmark LIS ISLR on real-world data. Baseline evaluations of the SignIT dataset demonstrate that current state-of-the-art approaches struggle to effectively classify LIS signs at the gloss level, as opposed to the categorical level [2]¹.

Zero-shot, few-shot and cross-lingual recognition

Early Zero-Shot SLR attempts struggled due to cross-lingual complexities between signs and natural language, as well as high variation in sign execution [10], [11], resulting in a pivot towards few-shot, visual retrieval paradigms [4].

Bilge et al. introduced Few-Shot Sign Language Recognition (FSSLR) via a meta-learning framework across sign languages, proving sparse source examples can generalise to unseen target languages. They discovered “synonym” subsets between languages failed to yield higher performance, suggesting signs are heavily diversified rather than net-iconic [4].

Similarly, Vandendriessche et al. (2025) embedded pose key points for distance-based visual retrieval, enabling one-shot ISLR that generalises to out-of-domain vocabularies without any retraining. Both frameworks operate entirely within a visual domain; achieving high cross-lingual transferability, but lack any inherent coupling to natural language text or semantic meaning [8].

In contrast, Cheng et al. utilise contrastive learning in CiCo to model retrieval as a cross-lingual problem, successfully aligning a single sign language video modality directly to a spoken language text space (e.g., ASL to English). It trains a domain-agnostic sign encoder before the domain-aware retrieval. [12].

SignCLIP, multilingual corpora, and multilingual sign language

SignCLIP aligns multilingual signs to a single text space in English (as a matter of efficiency). Their work relies on the ‘Iconicity Hypothesis’ - that universal motion primitives are semantically shared across sign languages, and adapts the distributional hypothesis to sign language. The model captures the core meaning of a sign as a ‘cluster centre’ in the embedding space, preserving the individual variance of different signers. However, mapping these clusters is made difficult by the Spreadthesign corpus, which is skewed to only one video per sign per language [1].

SignCLIP uses cross-lingual contrastive learning with prefixed language identifying tokens, e.g ‘<en> <ase> {word}’ for ASL. Ultimately, the authors note that the model’s zero-shot performance on out-of-domain data is deficient, and they posit that few-shot learning or fine-tuning is necessary to achieve noticeable performance [1].

The authors do not investigate the underlying architectural or semantic mechanisms that cause this failure, leaving the specific limitations of their cross-modal alignment unexamined.²

Datasets

A3LIS-147 characteristics

• Scale and composition: The dataset comprises exactly 1,470 (1490) video sequences representing 147 (+2) distinct, isolated LIS signs.
• Semantic categories: The vocabulary is distributed across six daily-life domains where automatic tools for social inclusion of deaf people could be effectively applied: Public Institute (39), Railway Station (35), Hospital (19), Highway (8), Common Life (16), and Education (30). There are 2 extra signs corresponding to the silence pose and the sign for “lingua dei segni”. We did not get access to the exact category matching for the vocabulary.
• Participants: Recordings were performed by 10 different signers (7 males, 3 females), each executing every sign in the vocabulary exactly once.
• Recording environment: Data collection was conducted in a highly controlled laboratory setting featuring a fixed frontal camera, a green-screen background, and uniform diffuse lighting provided by two 400W spotlights [5].

SignIT characteristics

• Scale and composition: The dataset comprises 644 video sequences representing 94 distinct, isolated LIS signs. The total collection spans approximately 3 and a half hours and contains around 99,000 annotated frames.
• Semantic categories: The vocabulary focuses on fundamental concepts that are typically taught first to sign language learners. It is divided into five macro-categories: Animals (32), Food (20), Family (20), Emotions (5), and Colours (17).
• Distribution: The dataset is long-tailed, with mean examples per class of 3.34 (train), 2.00 (test), and 1.50 (validation). The mode is 2 (train), 1 (test), and 1 (validation). 45 signers contribute unevenly.
• Recording environment: In contrast to the highly controlled green-screen setting of A3LIS-147, SignIT was collected from publicly available online sources, including YouTube user profiles, providing a wide variety of backgrounds and lighting conditions. Video resolutions range from 426x240 to 1024x1024 pixels at 24 to 30 frames per second.
• Preprocessing details: Some videos contained background text of gloss labels; to ensure models learn from the signer’s movements rather than background information, the authors applied a blur to those areas³ [2].

Preprocessing

We follow the same pipeline used for the training of the frozen SignCLIP backbone [1], including:

• 3D keypoint extraction: Trimmed RGB videos are processed through the MediaPipe Holistic framework to extract 3D spatial coordinates (x, y, z) of the signers per frame.
• Pose preprocessing: MediaPipe’s output is filtered to align with the SignCLIP video encoder. Lower-body keypoints are discarded, and the face mesh is simplified to core contours, reducing the feature space to 203 relevant keypoints per frame.
• Sequence normalisation: For signer-invariant representations, we spatially normalise the skeleton by shifting the shoulder midpoint to the origin and scaling all coordinates by the shoulder width.
• Truncation and padding: The resulting sequence is truncated or zero-padded to a fixed context length of 256 frames to match the required input dimensions of the SignCLIP architecture. For A3LIS-147, 10 videos were truncated, with the longest one being 269 frames. For SignIT, 39 videos were truncated, the longest being 739 frames. Whilst this is not ideal, we notice that the longer videos of SignIT are typically due to a repetition of the sign motion.

Methodology

We investigate whether SignCLIP’s multilingual pretraining generalises to LIS, a language present in the pretraining corpus but excluded from the original evaluation. Our approach tests this through three phases: Zero-shot evaluation, to assess the frozen multilingual prior’s native LIS structure; few-shot adaptation, to evaluate whether this structure supports recognition from a minimal number of examples using a frozen backbone; and a fine-tuning ablation, to determine the performance ceiling of lightweight fine-tuning whilst preserving cross-modal alignment.

Dual-dataset evaluation

A3LIS-147 and SignIT together evaluate model transfer across three axes:

• Vocabulary coverage: A3LIS-147 tests institutional language, which likely does not benefit from sign iconicity, and tests out-of-vocabulary (OOV) transfer with 31% of its signs. SignIT assesses in-vocabulary transfer with 95.6% coverage on foundational vocabulary, most likely to benefit from sign-iconicity.
• Acquisition conditions: A3LIS-147 conditions are likely to provide clean keypoint extraction matching the pretrained conditions. SignIT is likely to have more variation, thereby assessing robustness to pose estimation noise, scaling, and signer variance encountered in real-world deployments.
• Sample distribution: A3LIS-147 evaluates optimisation under balanced conditions, while SignIT approximates more difficult one-to-three-shot scenarios.

Zero-shot evaluation

The zero-shot evaluation applies the frozen SignCLIP checkpoint directly to both datasets. Predictions are generated by computing the cosine similarity between the video embedding and the text embedding of the English gloss (prompted as <en> <lis> [gloss]).

We report Recall@1, 5, 10, and Median Rank. To better investigate the “Iconicity Hypothesis” and transfer ability, we perform a per-class analysis stratified by Category, Median Rank, qualitative ASL/BSL similarity (iconicity proxy), and Spreadthesign presence.

Translation. We manually translated A3LIS-147 using Spreadthesign. Remaining out-of-vocabulary (OOV) terms were translated as accurately as possible. We also recreated the unavailable categories. Both are listed in Appendix D.

Few-shot evaluation

We evaluate few-shot ISLR to determine whether the solid results reported in the SignCLIP paper generalise to LIS.

• Linear probing: Following the original SignCLIP protocol [1], we train a scikit-learn linear classifier with 100 iterations on embeddings from the frozen backbone to evaluate the linear separability of the embedding space.
• Prototypical retrieval: The original SignCLIP few-shot evaluation also uses a $𝑘$-NN classifier ($𝐾=\lfloor \sqrt{𝑁}\rfloor$), but yields poor performance across benchmarks [1]. We instead adopt the Prototypical Network framework [14]. By computing a mean prototype for each class from its support embeddings and performing retrieval via nearest distance, we mitigate the impact of outliers. Unlike linear probing, this non-parametric approach naturally supports open-vocabulary classification without requiring retraining.

Fine-tuning and loss function ablation on A3LIS-147

We initialise from the baseline checkpoint and fine-tune on A3LIS-147 using a 70/10/20 signer-stratified split. This ensures that our evaluation measures generalisation to unseen signers (see Appendix D for the exact partition). Each configuration is trained for 50 epochs and evaluated across zero-shot retrieval, linear probing, and prototypical retrieval. For all the details about the hyperparameters used, see Appendix B.

The text Transformer ${𝑓}_{{𝜃}_{𝑡}}$ and CNN backbone ${𝑓}_{{𝜃}_{\text{CNN}}}$ are frozen to preserve pre-trained semantic anchors. We unfreeze the visual adaptation parameters ${\mathrm{\Theta }}_{\text{adapt}}=\left\{{𝜃}_{\text{MLP}},{𝜃}_{𝑣},𝜏\right\}$, denoting the video token MLP, video Transformer encoder, and logit-scale temperature, respectively.

Because contrastive models exhibit high sensitivity to objectives and batch scales on low-resource datasets, we conduct an ablation evaluating the following optimisation regimes:

• InfoNCE: Used during SignCLIP’s pretraining. It treats all other cross-modal pairs within the current batch as negatives [15].
• Supervised Contrastive (SupCon): Pulls together all same-class samples in the batch, whilst pushing apart samples from different classes [16].
• Cross-Entropy (CE): Focuses strictly on class discriminability over cross-modal alignment.
• Decoupled Hard Negative Noise Contrastive Estimation (DHN-NCE): Alleviates the negative-positive coupling effect that suppresses gradients from hard negatives when positive pairs are already well-aligned [17].
• Global Noise-Contrastive Estimation (GlobalNCE): Draws negative instances from a noise distribution over the full dataset, effectively decoupling negative sampling from the mini-batch size [18].
• ProLIP: A parameter-efficient regime that isolates adaptation to a minimal subspace by only unfreezing the final linear layer of the projection MLP ${𝜃}_{\text{MLP}}$ and the logit scale $𝜏$ [19]. Cross-entropy training is regularised by penalising the weight divergence between the fine-tuned and pre-trained projection matrices, adapting the model without destroying foundational cross-modal structures [19].

SignIT fine-tuning

The single best-performing fine-tuning regime identified on A3LIS-147 is applied to SignIT (details in Appendix B). To address the dataset’s naturalistic acquisition and long-tailed distribution, we apply light spatial augmentation to preserve semantic meaning, and heavier temporal augmentation (aug_sigma_temporal: 0.25, aug_sigma_spatial: 0.15, aug_sigma_noise: 0.002, aug_p_flip: 0.0,aug_strength_max: 3.5). SignIT’s richer macro-categories, and its previous literature motived additional experiments on category zero-shot and few-shot retrieval. For these experiments, we include recall, precision, and F1 alongside R@1 for better comparison with the original authors.

Experiments

Zero-shot complete-dataset evaluations

Baseline zero-shot evaluations in Table 1 and Table 2 show poor overall performance, in line with Jiang et al. findings for out-of-domain transfer [1]. However, there is stratification between categories. In SignIT, the ‘Food’ domain achieves the highest exact retrieval (10.96% R@1), while ‘Emotions’ demonstrates superior neighbourhood alignment (51.60% R@10). Similarly, A3LIS-147 exhibits a split between early recall (‘Common Life’, 7.22% R@1) and broader neighbourhood density (‘Public Institute’, MedR 42.5). This variance indicates that while overall cross-lingual transfer is weak, the model successfully transfers universal, cross-lingual iconic primitives from the pretraining distribution for specific semantic clusters. For gloss-level and more category details, see Appendix C.

Zero-shot medianK stratification, iconicity, and OOV analysis

SignCLIP’s cross-lingual alignment induces a structurally bimodal transfer effect. We argue that since the pre-trained text encoder operates in an English-centric semantic space, language prefix identifiers provide insufficient separation. Consequently, the objective forces visually disparate sign videos toward a quasi-singular text anchor. This semantic asymmetry creates an optimisation conflict that marginalises low-resource languages, resulting in negative transfer, evidenced by the adverse tiers in A3LIS-147 (24.16%) and SignIT (14.0%) in Table 3 and Table 4.

For iconic signs, the shared anchor is beneficial (achieving a MedR of 42.0); for non-iconic signs, the anchor provides a weak or adversarial signal, collapsing retrieval accuracy (MedR 60.1) in Table 6. Pre-training exposure does not overcome this issue, Table 5 shows OOV LIS signs marginally outperform in-vocabulary signs at R@1 (4.13% vs. 3.37%).

We believe data scaling is unlikely to resolve these failures. Shared human articulatory constraints result in heavy overlap in the discriminative features between languages, a problem further complicated by high individual signer-variance (Figure 1 in Appendix). Thus, diversification within synonym classes [4] and cross-lingual “false friends” lead to gradient conflicts. Our findings suggest these factors limit zero-shot performance for any architecture imposing a single joint embedding space without language-gated alignment. These issues can be resolved by monolingual fine-tuning (Table 8), likely at the expense of multilingual understanding, but this remains unexamined.

Linear probing on the frozen backbone achieves 66.78% R@1 on A3LIS-147 in Table 7, confirming that the video encoder learns robust representations.

A3LIS fine-tuning ablation

Table 7 shows that GlobalNCE yields the strongest fine-tuning performance on A3LIS and the linear-probe matches previous SOTA [6]. We attribute this to its global negative sampling across distributed batches, providing the critical density of hard negatives required to stabilise contrastive gradients. ProLIP achieves within 0.3% R@1 of GlobalNCE at zero-shot (75.84% vs. 76.17%) while adapting only the final MLP layer and logit scale, making it the preferred regime when compute or overfitting risk is the primary concern.

SignIT few-shot and fine-tuning ablation

Table 8 shows that augmentation of SignIT improves generalisation. Our results trail the LLaVA-OneVision (Acc 0.238 video+pose) of the SignIT authors [2]. We outperform all non-video baselines they evaluated, including pose-only LLaVA (Acc 0.121), establishing a competitive key point-only result.

SignIT macro-category retrieval

Zero-shot on categories achieves an F1-score (0.48) that is competitive with some fully supervised video baselines, such as I3D (0.34 F1)[2]. Because this relies on measuring the distance between visual embeddings and the textual embeddings of broad macro-categories, these results highlight an advantage of contrastive pretraining: the latent space is semantically organised, allowing the model to generalise to categorical distributions it never explicitly encountered during pretraining. Our strongest few-shot linear-probe configuration reaches 64.62% R@1, approaching the performance of SignIT’s best fully supervised MLP (0.726 Accuracy) [2].

Sign language identification

The Sign language identification of Table 10 complicates our earlier finding that in-vocabulary LIS signs do not outperform OOV. This simplified retrieval task suggests that SignCLIP does learn some language separation, as shown by the R@2 (65.23%). However, performance drops sharply at R@1 (35.10%), with substantial confusion between LIS, BSL, and NGT (Appendix A.3). It may be worth investigating if this is due to higher inter-language iconicity.

Conclusion

This work demonstrates that SignCLIP’s contrastive alignment induces a structurally bimodal transfer effect on LIS, beneficial for iconic vocabulary, adverse for non-iconic signs, indicating a geometric limitation of the shared embedding space paradigm rather than a data-scaling problem. Few-shot and fine-tuning strategies mitigate these limitations, confirming that the video encoder learns discriminative representations that zero-shot retrieval cannot exploit without fine-tuning in a monolingual context.

We see two promising directions for future research. Since pretraining exposure to LIS signs does not guarantee positive transfer, fine-tuning on the LIS-specific Spreadthesign subset could be adequate for OOD LIS. A more effective multilingual embedding space requires language-conditioned projections that both allow for iconicity transfer and decouple text anchors for non-iconic glosses across sign languages.

A.1 Leave-one-signer-out baseline linear-probe

Signer variability presented in Figure 1 primarily degrades R@1, seen by its $\pm$6.3% standard deviation. Broader retrieval remains robust. This variance underscores cross-signer generalisation as a persistent difficulty.

A.2 Complete A3LIS fine-tuning ablation

In Section 4.5, we presented a condensed view of our A3LIS-147 fine-tuning ablation, highlighting the performance of the default SignCLIP objective (NCE) against our best-performing GlobalNCE regime. Table 11 presents the comprehensive results across all evaluated loss functions, batch sizes, and sampling strategies.

A.3 Sign language identification scores

B.1 SignIT with augmentation fine-tuning hyperparameters

B.2 A3LIS and no augmentation fine-tuning hyperparameters

Note for ProLIP, there are two additional hyperparamters set: prolip_lambda: 0.5, and prolip_lambda_mode: inv_n

C.1 SignIT glosses by median rank

1. Great (1-3): bear, bread, color, watermelon.

2. Good (3.1-10): anger, brown, cake, chocolate, cow, fear, fuchsia, giraffe, grey, joy, light colors, orange, pizza, relatives, rooster, salt, sheep, snail, tiger, vegetable, wine.

3. Fair (10.1-25): apple, banana, bird, blue, butterfly, candy, cat, dark colors, disgust, donkey, family, fish, frog, fruit, grandfather, green, horse, light blue, lion, meat, monkey, parents, pasta, pear, pig, pineapple, pink, purple, rabbit, rice, spider, turtle, yellow, zebra.

4. Neutral / Random (25.1-47): aunt, black, brother-in-law, bull, cousin, crocodile, dad, daughter-in-law, dog, elephant, goat, goose, grandmother, milk, parrot, red, sadness, sky blue, uncle, water, wolf.

5. Perverse (47.1-93): boyfriend, brother, hen, husband, mom, mouse, nephew, sister, snake, son, son-in-law, white, wife.

C.2 A3LIS-147 glosses by median rank

1. Great (1-3): caldo, data, falconara, freddo, giudizio, iniezione, scadenza, senigallia.

2. Good (3.1-15): abitare, affitto, ancona, aperto, avviso, consegnare, dirigente, dolore, emergenza, jesi, macerata, modello, modulo, multa, notte, pomeriggio, presente, pubblica, ritirare_il_numero, sciopero, sostegno, traffico, tratta, vacanze, verde.

3. Fair (15.1-40): acqua, allegare, ambulanza, annullato, arrivo, ascoli, banca, binario, cambio, commissione, compilare, costo, cura, domenica, esame, fermo, giallo, giovedì, giorno, infermiere, infezione, istituto, marche, mattina, medico, operazione, partenza, promosso, provincia, ritardo, s.benedetto, tassa, torino, università.

4. Neutral / Random (40.1-74): abbonamento, allergia, amministrazione, andata, andata_e_ritorno, assente, assistente_alla_comunicazione, bidello, biglietto, bocciato, casa, casello, chiuso, cibo, civitanova, comune, diploma, disinfettare, fano, venerdì, giorni, ieri, laurea, litro, lunedì, martedì, mercoledì, mesi, obliterare, ospedale, pesaro-urbino, posta, rallentamenti, regione, ricevuta, ritorno, roma, rosso, segretario, sera, sindaco, stazione, strada, treno.

5. Perverse (74.1-148): asilo_nido, assessore, assistente, autostrada, domani, elementari, ente_pubblico, entro, flebo, impiegato, interprete, lingua_dei_segni, malattia, mangiare, marca_da_bollo, medie, nota, oggi, obliteratrice, orari, preside, professore, pronto_soccorso, registro, sabato, sala_d’attesa, scuola, scuola_materna, sil, superiori, sportello, studente, tecnico, telefono, ufficio_informazioni, voto.

C.3 SignIT median-rank category proportions

C.4 A3LIS-147 median-rank category proportions

The following table provides the full mapping used for our A3LIS-147 analysis, including category classification, presence in the SpreadTheSign (STS) corpus, and our qualitative iconicity proxy (visual similarity to English-speaking sign languages).