My external HDD acts super slow sometimes on my Series X

I bought a Western Digital 4TB HDD for my Xbox Series X and the thing is ridiculously slow at times. I started playing Wreckfest and it took like 3 minutes to load a game. Sometimes it’s fast and it loads normally as a game would on an HDD, like it did on my One and One X.

A couple of games can randomly take up to 4 minutes sometimes to load. 4 DAMN MINUTES. Is this happening to anyone else? Is there an specific issue that I’m missing or something? It is super random. I tried Mass Effect and it was fast (30 seconds maybe) but after turning the console off, it took 3 minutes to load a save file on the next session.