Are there any specific times of day when the game loads faster?
Absolutely. Try gaming outside of peak hours. In the UK, that means weekday mornings or afternoons. Friday night or Sunday evening are commonly the worst times, when everyone logs in. Fewer players online generally means less strain on the servers and a quicker load for you.