TraceOpt blog

Notes on PyTorch training performance

Latest

Training diagnostics July 14, 2026 8 min read

Same BERT run, three GPUs

I moved the same BERT-large training recipe across an L40S, A100 SXM, and H100 SXM on RunPod. The raw throughput ranking was expected. The useful part was the TraceML fingerprint: input wait and H2D copy were negligible, all three runs were compute-bound, and the H100 result pointed to a simple batch-size follow-up.

Read the post

Fixed batch

385.8

H100 samples/sec at batch 32

Follow-up

487.4

H100 samples/sec at batch 64

Signal

91.4%

GPU utilization after increasing batch

Try TraceML

Run it on one real training script.

The fastest way to understand TraceML is to run it on a job you already know and check whether the fingerprint matches your intuition.

If the summary is useful, tell us what it showed. If it is wrong, confusing, or does not handle your setup, please open an issue.

View on GitHub Read docs