I have a rooted S2 which I often use for watching anime. Let me just say: you won't be able to play 1080p 10bit smoothly. 480p will run smooth both 8bit and 10bit, 720p 8bit is also smooth, 720p 10bit will lag a bit but if you overclock it is watchable. But i'd recommend against overclocking because it just makes the phone really hot and it will drain your battery very fast.
Also, the phone screen is just 800x480. So if your video is over 800x450 it will be downscaled to match the screen, anyways. I've found that the best solution is just to reencode the series I want to watch into 800x450 with handbrake. You will save space and the quality will be practically the same, depending on your reencode settings. With a quality of RF19 and a few other settings I cannot even tell the difference between the original 720p and the reencoded 800x450 one. Reencoding takes about 10 minutes per episode on my 4 year old computer so it isn't such a hassle, either.
As for players, just go with MX Player. I have tried about all of them and it is the best one out there. BSPlayer gets a lot of praise but I found its settings far more limited than those in MX and the performance was mainly the same. MX has yet to fail me in displaying any softsubs, so that's also a plus.
I, myself am not yet using jelly bean, mainly because i'm lazy and partly because the ICS roms just work and are super stable. It doesn't help that JB lacks a native flash support and it doesn't have any particular performance improvements. Just run a light ICS rom and you should have maximal performance and stability.