If you are doing NAT on a connection, all packets passing both ways (in and out of the network) must pass through the NAT'ed box, otherwise it won't work reliably. In particular, the connection tracking code reassembles fragments, which means that not only will connection tracking not be reliable, but your packets may not get through at all, as fragments will be withheld.