The Youtuber Brandon F has a 4 part series talking about why they fought like this. Spoiler- it wasn’t because they were stupid.

Part 1

TLDR- if you split up you just get run down by enemy cavalry.

Part 2

TLDR- a close formation lets you concentrate your firepower at one point.

Part 3

TLDR- a close formation makes communication and controlling the army much much easier (or even possible at all).

Part 4

TLDR- the formation makes the troops less likely to run away.