Why does a helium balloon fly?

Now you know how much lifting force is generated. To make clear, why buoyancy is generated, we consider the atmospheric pressure.

Gravity makes sure that the atmospheric pressure is greatest on the ground. The higher you go, the lower is the atmospheric pressure.

So the atmospheric pressure is lower on the upper part of the balloon than on the lower part. Therefore the balloon is affected by a greater force from the downside than from the upper part. The difference of these two forces is the buoyancy.

But the balloon is affected by the atmospheric pressure from all sides. Thus there are lateral forces acting on the balloon. But why do we not have to consider them?