From my experience, it is not always just the network. Ansible can be slow in operations such as copying files between directories on the remote target. I suffered from this a couple of years ago and the solution I implemented was to delegate the copy operation to a bash script and leave Ansible to just take care of uploading that script and run it passing the configured parameters.

Anyways, if the network is a problem I strongly recommend to have a look to "Ansible pull".