Why Football is so Difficult to Referee

Just a brief thought which is pretty obvious, really, but doesn't get said enough. What makes football, and also, to a lesser extent, rugby, so difficult to officiate as opposed to many other sports is how often the referee is expected, in making a judgement, to read intention. A player can be sent off or unpunished depending on what a referee believes, from what he sees his body do, is in his mind.
At the speed at which football is played, bearing in mind we're talking quite often about actions carried out with the feet rather than the hands, this is an unreasonably difficult task.