This might sound a bit crazy, but try drilling slowly backwards with a hand drill. The action is more like filing and you don't get the drill bit digging in and tearing the wood.