Hugging Face Trainer integration
Find why your Hugging Face training run is slow.
TraceML adds step-level bottleneck diagnosis to the Trainer workflow you already have. It separates input wait, transfer, compute, and optimizer time—without a manual training loop.
Same Trainer. Better-fed GPU.
The notebook trains ResNet-50 twice on full-resolution Imagenette. Only the data-loader settings change.
The run finished 1.83× faster after the input wait was removed on a free Colab T4.
Keep your Trainer. Add two small changes.
Initialize TraceML once so it can time the Trainer-created DataLoader and step phases. Then use the TraceML Trainer drop-in in the place you already create your Trainer.
from traceml_ai.integrations import \
huggingface as traceml_hf
traceml_hf.init()This enables DataLoader fetch, host-to-device, forward, backward, and optimizer timing around each Trainer step.
trainer = traceml_hf.TraceMLTrainer(
model=model,
args=training_args,
train_dataset=train_dataset,
traceml_enabled=True,
)
trainer.train()Run with traceml run --mode summary ... and TraceML returns a direct diagnosis plus a portable final summary.
Measure the before and after on your own hardware.
The Colab changes only three TrainingArguments data-loader settings: worker count, pinned memory, and persistent workers. It keeps the model, dataset, batch size, transforms, and step count fixed.
What changed?
The GPU stopped waiting for decoded batches. The exact speedup depends on your CPU cores, storage, transforms, and model cost; the notebook is designed to show the answer for your own run.
Fix the bottleneck TraceML actually finds.
Low GPU utilization alone cannot tell you whether the loader, device transfer, or model is responsible. The step-time breakdown makes the next experiment concrete.
Input wait
Test workers, persistent workers, pinned memory, transforms, collation, and storage only when input wait is material.
H2D transfer
Separate host-to-device transfer overhead from a data pipeline that cannot produce batches quickly enough.
Model compute
When the verdict is compute-bound, more loader workers will not make the job faster. Focus on the model or step cost.
Start with the runnable proof.
Open the notebook, run both profiles, then carry the same integration into your existing Hugging Face script.