It's not your job to try to get the casting director to see all the other characters. They know the scene, they've probably read it many times already. Your job is to show the casting director that you are the character and the best way to do that is by looking at them and establishing a connection with them. So put all the characters in the eyes of the casting director.
Now you can definitely suggest different relationships with the different characters through your body language, intentions, thoughts, etc. But you don't need to look away to do that.
Even if you think it helps you be in the scene by putting characters all over the room, what little you gain by doing that, you more than lose because you've lost the connection with the casting director.
If you absolutely must look away for some reason, make it 10 or 20 percent of the scene, not 80 or 90 percent!
You may feel the casting director isn't giving you much. But they are giving you something, sometimes they're giving you a lot. So take the gift of their presence and use it. Play with the casting director instead of the imaginary characters that exist only in your mind!