Some anti-Tank guns and Tank Destroyers did indeed fall under Tank Destroyer Branch, later Command.
In terms of anti-tank guns, it's my understanding that the 37 MM and 57 MM remained under infantry branch control, but the larger 3 inch AT guns, and the 90 MM had it been adopted would have belonged to Tank Destroyer units.
In terms of artillery, actually the tank destroyer branch was the brainchild of artillery officers, and many of its initial cadre of men were artillerymen given the skillset behind operating an AT gun.