It's not the weather as much as it is likely a combination of prep and the cold asphalt. To do it right, you want to cut the pothole out square and use hot mix. I've never had the stuff from Home Depot last more than a few weeks just pouring it into a hole and tamping it down, even when it's 110? out.
I use Sakrete instead of Quikrete, my local True Value hardware store carries it. I like the Sakrete that is like a brick when you buy it. I take a big torch(roofing) and heat it up in a metal wheelbarrow before spreading and tamping. I think HD sells Quikrete. The Quikrete stuff that just pours out of the bag won't solidify fast enough at low temps.
I do a combination of hole prep and heat. Cut the hole out as square as possible. I put the quikrete cold patch in a wheel barrow and heat it up with a torch for a good 10 minutes. I then heat the edges of the hole and the base so that the ground doesn't suck the heat out too quick. Then I shovel in the cold patch and hit it with some more heat. I then hand tamp the whole thing and use some asphalt sealant around the edges since that seems to be where it wants to come up anyway. I let it set for a day with some cones around it and then I am good to go. It has always worked well for me.
Here's an idea I stole from the town I live in. They cut out the pot hole (like a few feet square), dig it out some, fill with gravel and or sand, then put in some paving stones, then sweep some sand in the cracks to set. It works very well, especially in the middle of winter when cold or hot patch is difficult.