Any load will affect the handling of your motorcycle by changing its centre of gravity. When using panniers, spread the weight evenly on each side. Avoid carrying heavy items in a top box, as this could make your steering dangerously light.
