Same BERT run, three GPUs

I wanted to run a small, controlled experiment: take the same PyTorch training script, keep the training config fixed, and move it across three GPUs on RunPod.

I am treating this as an untuned baseline, not as a full GPU benchmark. In an MLPerf-style benchmark, each system is tuned carefully so the final numbers represent best-case standardized performance. I wanted the opposite starting point:

The code is in the TraceML repo: examples/advanced/bert_single_gpu_compare.py. It is a simple Hugging Face BERT text-classification loop on AG News, using PyTorch AdamW, a linear learning-rate schedule, and mixed precision.

I started with BERT-large on a single GPU. The fixed config was batch size 32, sequence length 256, fp16, AdamW, learning rate 2e-5, 1,100 total steps, and the first 100 steps excluded from measurement. I ran the same script on L40S, A100 SXM, and H100 SXM.

Each run was wrapped with TraceML so I could see the step breakdown, not just the final samples/sec number.

After the first comparison, I ran one follow-up on the H100 with batch size 64. That was not a full tuning sweep. It was the next experiment suggested by the first TraceML summary.

The basic result

The raw throughput result was not surprising.

The H100 was fastest. The A100 was next. The L40S was slower. That is the expected part. The more useful part was what TraceML showed underneath the throughput number.

All three runs were compute-bound

TraceML gave the same primary diagnosis for all three runs: COMPUTE-BOUND. The input path was not the issue. Dataloader time, input wait, and host-to-device copy were all very small.

So this is not a dataloader tuning story. It is not a CPU bottleneck story. It is not a host-to-device transfer story.

For this workload and this config, almost all of the measured step time was model compute. That is already useful. It rules out several guesses quickly. If I had only looked at throughput, I would know which GPU was faster. I would not know that the input path was essentially out of the way.

Same verdict, different fingerprints

The interesting part is that the same COMPUTE-BOUND diagnosis did not mean the three runs looked identical. The L40S and A100 both showed high GPU utilization and high GPU power warnings.

The H100 was much faster, but at this fixed batch size it also had lower average GPU utilization: 84.8%, compared with about 93% on the A100 and L40S.

I do not read that as a dramatic result. It does not mean the H100 is bad, and it does not prove the job is poorly tuned. It does tell me something concrete: with this exact batch size and sequence length, the H100 has more headroom than the other two cards.

This lines up with the general point in NVIDIA's GPU performance docs. GPUs need enough parallel work to keep their SMs occupied. NVIDIA describes the tail-effect problem: if a kernel launch does not create enough waves of work across SMs, part of the GPU can sit underutilized near the end of the launch. The practical lesson is enough for this post: fixed-batch comparisons need a run-level fingerprint, because the same recipe can fill different GPUs differently.

Lambda's public GPU benchmark methodology makes a related practical point: training throughput depends on batch size, GPU choice, multi-GPU parallelism, and model implementation. Keeping the batch fixed answers one question:

What happens if I move the same recipe across machines?

It does not answer a different question:

What is the best tuned throughput each GPU can reach?

Both questions are valid. They just should not be mixed.

A small follow-up on the H100

The fixed batch-size run raised an obvious next question: if the H100 showed more headroom at batch size 32, what happens if I make the workload a little larger?

So I reran only the H100 at batch size 64, keeping the model, sequence length, precision, warmup, and measured window the same. I ran it three times. The samples/sec results were 487.4, 486.7, and 488.4, so I used the middle run as the summary number.

This is the part that made the experiment more useful to me.

The first TraceML summary did not just say "H100 is faster." It showed that, at batch size 32, the H100 had more system headroom than the A100 and L40S. Increasing the H100 batch size to 64 moved the fingerprint in the expected direction: throughput improved by about 26%, GPU utilization rose from 84.8% to about 91%, memory use increased, and TraceML started flagging high GPU power.

Input wait and host-to-device copy stayed tiny. So the next useful experiment was not dataloader tuning or CPU tuning. It was increasing the amount of work given to the GPU.

I still do not call batch size 64 "the tuned H100 result." Batch size 128 may move the numbers again, and in a real training run larger batch sizes can affect convergence and optimizer choices. The useful point is narrower and stronger: the first fingerprint suggested a concrete next experiment, and the follow-up moved the run in the direction the fingerprint suggested.

The phase breakdown also changed

TraceML also split the original batch-size 32 compute into forward, backward, and optimizer time.

Backward was the largest phase everywhere. But the shape of the step still changed across GPUs.

The optimizer here is plain PyTorch AdamW. That matters because the possible next steps depend on which phase is actually expensive. If optimizer time is tiny, changing optimizer implementation is unlikely to be the first place I would spend time. If optimizer time is a meaningful chunk of the step, then optimizer choices, fused implementations, or precision details may be worth checking.

On the L40S, optimizer time was 43.5 ms. On the H100, it was 11.4 ms. That does not mean "optimize AdamW" is the right answer for this workload, because backward was still the largest phase on every GPU. But it does show why I want the phase split. It keeps the next optimization discussion attached to the actual run instead of to habit.

This is the part I found most useful. A single throughput number tells me the H100 is faster. A step fingerprint tells me what kind of faster it is.

What I take from this

When I moved the same training job across three GPUs, the headline diagnosis stayed the same, but the run fingerprints were different. And when one fingerprint suggested a follow-up, changing one knob on the H100 moved the run in the direction I would have hoped.

TraceML made that visible in one summary:

• the run was compute-bound
• input wait was negligible
• H2D copy was negligible
• memory pressure was not the issue
• L40S and A100 were closer to their power limits
• H100 was much faster but showed more headroom at this fixed config
• increasing H100 batch size from 32 to 64 raised throughput and GPU utilization
• backward was the largest phase on every GPU

That is enough to decide what to do next.

For this specific run, I do not spend time tuning the dataloader. I do not chase host-to-device copy. The next experiment is a fuller batch-size sweep, especially on the H100, to see where throughput and utilization stop improving.

Why I care about this

A lot of training performance work starts with guessing. Maybe the dataloader is slow. Maybe the GPU is underused. Maybe the batch is too small. Maybe a larger GPU will fix it. Maybe it will not. Sometimes the guess is right. Often it is not.

The goal of TraceML is to make the first look cheap and repeatable. It is not trying to replace Nsight, PyTorch Profiler, or deeper kernel-level tools. Those are still the right tools when you need that level of detail. TraceML is meant to sit earlier in the workflow: Run the job. Get the step breakdown. See whether the bottleneck is input, transfer, compute, memory, optimizer, or system pressure. Then decide where to spend time.

For me, this experiment was a useful baseline. Same code, same batch size, same sequence length, three GPUs. The answer was not surprising, but it was clearer than a throughput table alone. The extra H100 batch-size run made it more useful: the summary did not just explain what happened, it pointed to a reasonable next experiment.

Try it

Run TraceML on one real training script.

TraceML is open source and early. If you run PyTorch training, especially on rented GPUs, try it on one real training script and see whether the fingerprint matches your intuition.

If the summary is useful, I would love to hear what it showed. If it is wrong, confusing, or does not handle your training setup cleanly, please open an issue. That feedback is the most useful thing right now.

View TraceML on GitHub Read docs

References

• NVIDIA GPU Performance Background
• Lambda GPU Benchmarks methodology
• MLPerf Training benchmark