Mixing 1/35 and 1/32 scale parts on the same figure would not look right. The difference is enough to be an issue as the head and say hands would not be in proper proportion.
Mixing 1/32 and 1/35 figures in the same dio is not really an issue though. People come in all sizes; tall, short, skinny, built, etc. The issue is with gear. As long as you keep all gear and accessories the same scale; for example, weapons, canteens, ammo pouches, you should be fine. As people do come in different sizes, gear is usually standard.
Placing them with 1/35 armor is not an issue either. Again, people come in different sizes. I often mix 1/35 figures with 1/32 aircraft, helos, and civilian vehicles. Likewise, I have mixed 1/32 figures with 1/35 vehicles as well. You can see some of them in my Showcases below.