Exploring In-Context Learning and Efficient LoRA Fine-Tuning of Gemma3 Models versus BERT Fine-Tuning in Low-Resource Environments

Article

all results can be found in the article.

Model Inventory

Family	Params (B)	Max Seq	Instruction-Tuned
Gemma 3-270m-it	0.27	2048	Yes
Gemma 3-1b-it	1.00	2048	Yes
Gemma 3-4b-it	4.00	2048	Yes
BERT base	0.11	256	No
DistilBERT base	0.06	256	No

LoRA Training Cost

Model	Domain	Dataset	Rank	Steps	Peak GPU (GB)	Time (h)
Gemma3-270M	CLS	AG News	128	500	4.8	0.2
Gemma3-270M	QA	SQuAD v2	128	500	9.9	0.4
Gemma3-1B	CLS	AG News	64	500	9.9	0.3
Gemma3-1B	QA	SQuAD v2	64	500	12.6	0.5
Gemma3-4B	CLS	AG News	32	500	12.6	1.0
Gemma3-4B	QA	SQuAD v2	32	500	12.9	2.7

Performance Results

Gemma3-270M Classification - accuracy

Model	k	AG News	SST-2	BoolQ	Mean
Base	0	0.33	0.52	0.56	0.47
Base	10	0.40	0.70	0.74	0.61
LoRA-CLS	0	0.72	0.66	0.40	0.59
LoRA-CLS	10	0.81	0.67	0.41	0.63
LoRA-QA	0	0.26	0.51	0.51	0.43

Gemma3-270M Question Answering - exact match

Model	k	SQuAD v2	TriviaQA	NQ-Open	Mean
Base	0	0.13	0.07	0.03	0.08
Base	10	0.10	0.07	0.06	0.08
LoRA-CLS	0	0.05	0.001	0.01	0.02
LoRA-QA	0	0.45	0.001	0.02	0.16
LoRA-QA	10	0.45	0.003	0.01	0.15

Gemma3-270M Reasoning - accuracy

Model	k	HellaSwag	ARC-Easy	PIQA	Social IQa	Mean
Base	0	0.39	0.51	0.67	0.42	0.50
Base	10	0.39	0.57	0.67	0.47	0.53
LoRA-CLS	0	0.35	0.52	0.66	0.38	0.48
LoRA-QA	0	0.31	0.46	0.60	0.39	0.44

Gemma3-4B Performance (Base Model Only) - accuracy

Task	k	Performance	Improvement vs k=0
Classification (Mean)	0	0.75	-
Classification (Mean)	10	0.87	+12%
QA (Mean EM)	0	0.21	-
QA (Mean EM)	10	0.34	+13%
Reasoning (Mean)	0	0.70	-
Reasoning (Mean)	10	0.74	+4%

BERT performance

Task	Dataset	Accuracy	F1
Classification	AG News	0.937	0.933
Classification	SST-2	0.486	0.152
Classification	BoolQ	0.398	0.260
QA (DistilBERT)	SQuAD v2	0.67 EM	0.71
QA (DistilBERT)	TriviaQA	0.01 EM	0.01

Inference efficiency

Latency comparison

Model	Adapter	k	TPOT (ms)	Throughput (tps)	P50 Latency (ms)
Gemma3-270M	Base	0	50.3	19.90	155.3
Gemma3-270M	Base	10	65.6	15.24	202.0
Gemma3-270M	LoRA-CLS	0	49.4	20.23	152.5
Gemma3-270M	LoRA-CLS	10	57.1	17.52	176.2
Gemma3-4B	Base	0	86.1	11.62	265.4
Gemma3-4B	Base	10	94.4	10.60	291.1
BERT	-	0	0.3	3837.1	8.1

ICL latency overhead

Model	Task	k=0→k=10 TPOT Increase	k=0→k=10 Throughput Decrease
Gemma3-270M	CLS	+30% (50.3→65.6 ms)	-23% (19.90→15.24 tps)
Gemma3-270M	QA	+24% (47.3→58.5 ms)	-19% (21.13→17.10 tps)
Gemma3-1B	CLS	+12% (68.5→77.0 ms)	-11% (14.60→12.98 tps)
Gemma3-1B	QA	+21% (57.8→69.7 ms)	-17% (17.31→14.36 tps)
Gemma3-4B	CLS	+10% (86.1→94.4 ms)	-9% (11.62→10.60 tps)
Gemma3-4B	QA	+17% (79.8→93.5 ms)	-14% (12.53→10.70 tps)

Conclusions

1. In-domain gains vs cross-domain perfomance drops

LoRA fine-tuning improves in-domain performance substantially (72% vs 33% on AG News for 270M model), but this comes at the cost of cross-domain generalization. LoRA-adapted models show up to 20% degradation on reasoning tasks compared to base instruction-tuned models. Fine-tuning on one task domain actively degrades performance on other domains, including within the same task category.

2. ICL gains scale with model size, lost in LoRA finetuned models

Larger models benefit more from in-context learning. Gemma3-4B with k=10 achieves 12% classification improvement, 13% QA improvement, and 4% reasoning improvement over k=0. At this scale, ICL matches or exceeds smaller LoRA-tuned models on non-target tasks while maintaining generalization. Smaller models show inconsistent ICL benefits, suggesting minimum capacity requirements.

3. Efficiency affected by context length, and not by LoRA

LoRA adapters introduce negligible overhead (TPOT changes of -1.8% to -8.2%). In contrast, increasing ICL context from k=0 to k=25 raises TPOT by 26-40% and reduces throughput by 21-28%. BERT remains most efficient for single tasks (3837 tps vs 12-20 tps for Gemma models) but shows zero cross-task transfer.

4. BERT best at a single task

BERT achieves 93.7% accuracy on AG News, outperforming all Gemma variants. However, it performs near random on other classification tasks (48.6% on SST-2, 39.8% on BoolQ) and reasoning tasks, confirming no generalization without additional fine-tuning.