As harsh as it sounds, I'd agree with the referee in this case. I am a referee in Scotland and I a fairly strict to be honest. I like to be punctual and do things proper. Therefore, if you entered the field of play with permission from the AR and no permission from the Referee, you were correctly cautioned for it.