I find handbrake slow and clunky, get AviDemux instead.
Open up the file, under the Video toggle set it to the MPEG-4 AVC codec (x264), apply any filters you want (if it's an interlaced source, you might want to run it through a yadif de-interlacer; if it needs cropping you can set that, noisy run the noise filter, change the aspect ratio, etc, etc, etc). Then under configure you have the options for encoding style. If you have a set size in mind, then two a two pass by size. Otherwise do a two pass by average bitrate (high quality encoded films from a DVD usually end up between 1400 and 1500kbps, as a ballpark figure to start playing around with). The only thing I'd bear in mind with the two pass-size setting is that the size only applies to the video, the audio gets lumbered on top of that.
Don't forget to change the format and you have to specify that you want to save it as .mkv for some odd reason.