Andrej Karpathy sur Twitter : most common neural net mistakes(About) > 1) you didn't try to overfit a single batch first.
["if you can't overfit on a tiny batch size, things are definitely broken"](https://youtu.be/gYpoJMlgyXA?t=1h1m22s)
> 2) you forgot to toggle train/eval mode for the net.
> 3) you forgot to .zero_grad() (in pytorch) before .backward().
>4) you passed softmaxed outputs to a loss that expects raw logits.
> 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters
> 6) thinking view() and permute() are the same thing (& incorrectly using view)