From the BCA Rule Book:

1.33 Disturbed Balls (Cue Ball Fouls Only) (AR)
1. It is not a foul to accidentally touch or disturb a single object ball, with any part of your body, clothing or equipment, unless the accidental movement has an effect on the outcome of the shot.
3. If there is no effect on the outcome of the shot, your opponent has the option to leave the disturbed ball in position or restore it to its original location on the table. If the disturbed ball is to be restored, a referee may restore it, your opponent may restore it, or you may restore it with your opponent’s permission. If you touch or restore the disturbed ball without your opponent's permission it is a foul.
7. It is a foul if you disturb more than one object ball.
8. It is a foul if a disturbed ball contacts any other ball.

VNEA - 8-Ball Rule:

8. ACCIDENTALLY moving or touching any ball is not a foul unless: 1) the moved ball is the cue ball or 2) a moved ball makes contact with the cue ball or 3) a moved ball that is jumped off the table or pocketed or causes any ball to be jumped off the table or pocketed. (exception to #3: If the 8-ball is jumped off the table or pocketed it is loss of game if called by the opponent before the next shot is taken). Only opponent may replace the ball moved as closely as possible or leave it where it rests. If the shooter replaces the moved ball, it will be considered a foul.

The BCA rule is: it would not be a foul unless the object ball was pushed into the cueball. Bridging over a ball and moving it by itself is NOT a foul.

Clearly a variety of other fouls could be committed but in the specific situation you described...its sounds like an accidentally moved ball...no foul.

No balls were moved, as I warned him they may move if he wasn't careful or moved the bridge adaptor to the end of the cue. A team mate of his said it wouldn't be a foul and I thought it would if he hit my object ball into the cue ball with the tip if a bar cue while using it as a bridge holder. I didn't want to have an issue last night and wanted to see what the actual rule was today.

So thanks for your input all.