Profiling TorchNeuron (Part 1): System-level profiling on AWS Trainium
This post teaches how to profile inference performance using AWS Trainium and TorchNeuron, a PyTorch backend for deep learning.
Intelligence analysis by Qwen 2.5 (3B)
Learn how to profile inference performance with AWS Trainium and TorchNeuron, a PyTorch backend for deep learning.
This post teaches how to use a special chip called AWS Trainium and a tool called TorchNeuron to see how fast a computer can run a deep learning model. It shows how to set up a tool to watch the model's work and see where it spends most of its time.
Analysis
Understanding the Profiler Setup
To set up the profiler, you need to configure NeuronConfig with the appropriate data streams and output directory. The modes parameter controls which data streams to capture, such as device instructions and runtime traces. The capture_enabled_for_nc parameter selects the NeuronCores to trace. Setting correlate_device_time to True propagates device time from the trace into the key_averages() table, where you can compare CPU and Neuron device time.
Profiling RMSNorm Operation
RMSNorm is a small operation with a clear sequence of arithmetic and reduction operations. The post profiles three implementations of the same operation: Naive RMSNorm with individual PyTorch operations, Naive RMSNorm with torch.compile, and the PyTorch built-in implementation. The results are compared to show the differences in performance and behavior.
Using Neuron Explorer
Neuron Explorer is a browser-based viewer for the system trace, device timeline, and per-engine breakdown. It can be accessed through an SSH tunnel or the Visual Studio Code extension. The post uses Explorer to confirm the numerical values reported by key_averages() and to inspect the sequence of events between dispatches.
Conclusion
By the end of the post, you should be able to configure the profiler with the Neuron backend, capture a clean trace, and understand how to read the key_averages() table to identify where the time is spent in the inference process.
Key points
- Learn how to set up the profiler with the Neuron backend.
- Profile three implementations of the same RMSNorm operation.
- Use Neuron Explorer to visualize the profiling results.
By understanding where the model spends its time, developers can make the model run faster and more efficiently.
If the model spends too much time on certain parts, it might not run as fast as expected, and developers might need to optimize those parts to improve performance.
