I always found modal editing most valuable when editing and navigating through large bodies of text. I'm not quite sure that it makes sense on the command line, where you're rarely dealing with more than a few lines of text, and seldom need to rearrange those particular lines.
Nevertheless, I switched to zsh on Sunday, and with it, vi keybindings. It's getting better, but it still doesn't feel quite right.
I use vim almost pathologically, but I gave up on set -o vi. Since there's no visual indicator of which mode you're in, it's easy to make confusing mistakes and not the right feedback. It just wasn't worth it.
I use vi mode in my shell all the time, and love it. It only took a day or so to get used to it. But then again, I am a die-hard vim user.
What I'd really like, though, would be a vim mode for the shell, not just a vi mode. Then I could use some of vim's more advanced features, which would make the mode even more useful.
I can only speak for myself. Give it a day or two, and it will quickly become as transparent as your editing sessions. Once you cut over, you can forget about chording and arrow keys. Just one key: <Esc>.
I found myself using Emacs keybinding for line wise operation and movement in both vim and shell. It is just verbose to switch between editing modes sometimes.