Hello
As far as I know still there is not direct bus lines between Dubrovnik and places in Montenegro, to go directly in Kotor it is possible only by organized coach tours, renting a car or by private taxi. Kotor is a nice town and it is worth to be seen (Boka Kotorska Bay as well). Crossing the border is very easy and you will not experience any problems. If you are interested to go there by private taxi contact me by e-mail.
Have a nice stay in Dubrovnik and Croatia
Best wishes
Tom