chunked/content_aware_chunking/_debugging/chunk_27.txt

In such a case you can use the
detect_overflow helper function to inject the detector where you want it, for example:
thon
from debug_utils import detect_overflow
class T5LayerFF(nn.Module):
    []
def forward(self, hidden_states):
    forwarded_states = self.layer_norm(hidden_states)
    detect_overflow(forwarded_states, "after layer_norm")
    forwarded_states = self.DenseReluDense(forwarded_states)
    detect_overflow(forwarded_states, "after DenseReluDense")
    return hidden_states + self.dropout(forwarded_states)

You can see that we added 2 of these and now we track if inf or nan for forwarded_states was detected
somewhere in between.
Actually, the detector already reports these because each of the calls in the example above is a nn.Module, but
let's say if you had some local direct calculations this is how you'd do that.
Additionally, if you're instantiating the debugger in your own code, you can adjust the number of frames printed from
its default, e.g.:
thon
from transformers.debug_utils import DebugUnderflowOverflow
debug_overflow = DebugUnderflowOverflow(model, max_frames_to_save=100)

Specific batch absolute min and max value tracing
The same debugging class can be used for per-batch tracing with the underflow/overflow detection feature turned off.
Let's say you want to watch the absolute min and max values for all the ingredients of each forward call of a given
batch, and only do that for batches 1 and 3.