Miles do matter but it's like everything - if you buy a 20k miler for £30k and run it to 50k, you'll lose £10,000 and have the same running costs. Whereas if you buy a higher miler, do the same mileage, so 66k to 96k, you should still see £12-14K for it at today's prices. So, you'd lose less money. Running costs would be similar IMHO as a low miler comes with it's own set of issues. For me, I bought Loz's Cat D for not much off that car but with a few valuable upgrades and I intend to use it regularly.
If you have it as a 2000 mile per annum 2nd car, I'd say you're better to buy a low miler and hope that it holds most of it's cost. If you want to use it, a mid-miler is better as fundamentally you have less depreciation to go therefore you should return more cash as a percentage of the purchase value.
I ran a 911 turbo that cost me £68K and had 15,000 miles on in the past. I sold it 2 years later at 30,000 miles for £41k. A big hit and if I was sensible, i should have bought an older car, below the £27K that I lost and not worried about what I would recover at sale time!
It's all subjective though as I also sold a gorgeous 993 turbo for £36k which would now be worth £55k! Who knows what's right?