- They’re not sewn at the hip.
- They still don’t like each other that much.
- Some things are just not anyone else’s business.
Probably having a life or doing things on their own while something is happening to the others?
Like Thor is in another realm while Iron Man 3 is happening in California/China (and the places Iron Man 3 takes place in) meanwhile Steve and Natasha along with Sam and Sharon are dealing with Bucky and other bad guys, while Clint is in a top secret mission for SHIELD and Bruce is walking around in an unknown region not wanting to be found.
Also they only assemble when there’s a super global threat that requires them to get together.