I have a feeling that CA will work best if applied straight to the fretboard. It will probably stick to the BLO at first, but might peel away with age. You'd probably need the BLO to be very well cured before doing so, which could take at least a month or more.

I'd apply a very light water or spirit stain to the fretboard first. You can always dilute it significantly so that you get very little tint on each application.