I often forget to use whatever skills I have equipped in the heat of battle, so lean more towards the permanent buffs on the weapon skill trees. The rest of the weapon-specific skill trees are more down to your personal preference.
There are four core skill trees that just about every character will use: Samurai, Shiftling, Ninja, and Onmyo Magic.