You left out a biggy....division rival loses league MVP 10 mins into the first quarter of game one.
You would not have won the division or made the playoffs without that.
MIA went 1-1 against NE in 2008. Do you really think Tom Brady was the difference between losing and winning the Wildcat game?
Perhaps you mean the other losses for which Cassel was at the helm. Those would be:
IND, SD, PIT, NYJ
Those are not exactly pushover teams (except NYJ, but you lost to them twice with a vastly superior team in 2010, so...)
For purposes of comparison, NE was 11-5 with Cassel in 2008 and 10-6 with Brady in 2009. The conclusion you have reached is not as foregone as you suspect.