Regarding blues and olives (and blacks) I used to do very well with these between San Francisco and Santa Cruz using light tackle and a dropper with a homemade white bucktail shrimpfly (+ mylar)tied onto a hook on which I pinned a motoroil grub. In the right areas these are almost "can't miss" species. Most were small blues, but I caught a 3.25 and and a 4.0 black on this rig.

I caught other rockfish and lings on the swimbaits there, but not blues or olives. You are right; you should probably go to smaller swimbaits for these species.

Good luck.