After drilling the 1/4" hole, use a sharp tool to score along the finished dimension.

Scoring along the edge will prevent unsightly breaking of the wood fibers.