You can try various settings, such as Stable Fit or higher quality shadows (this is really just 'use a bigger texture' so those jaggies get smaller as the same volume is covered by more shadow-pixels) but fundamentally you're not going to get a better result. This also explains why changing the light's position and orientation to a different angle affects the result: those edges more closely (or less closely) align with where the lightmap pixels are. Which means that some edge, somewhere, is going to get jaggy. And in fact, can never be as your cubes are rotated five to ten degrees from each other forming a curve. Your objects and light are definitely not square up from each other. This shadow texture is then 'cast' from the light source (think about the light as being a camera: every pixel that it can see that is an object, that becomes a 'dark' pixel in the lightmap its a bit more complicated than that, but not by much).

And texture points (aka 'pixels') are square.

The problem is that the shadows being cast are essentially just a texture.

How to uninstall unity spotlight