You may have been adding friends the wrong way all along.
If you guys find it a pain in the ass to add friends to play customs with you may be going about it wrong. The 100% best way is to go on www.xbox.com sign in go to friends and use that to add mass people, and to check your friend requests and add them all in seconds go to followers and the requests will have a little + sign next to them and adding them is as easy as 1 click without even leaving the followers page. Hope this helped some people.
Another way that works pretty well is using the Xbox Smartglass app on your phone. Works the same as this just on your mobile. Definitely a must have beside you while gaming.