Rainbows appear directly opposite the sun with respect to the observer. Think of it this way: if you are standing with your back facing the sun, the rainbow should be right in front of you
This means that if you are not standing in front of the sun, you would not be able to see the rainbow! Consider the few situations when you would not be standing in front of the sun:
1) When there is simply no sun
2) When the sun is directly overhead - which is also why we usually see rainbows in the late afternoon when the sun is low in the sky
3) When you are in the wrong position
Nonetheless, do remember that we are assuming that atmospheric conditions are ideal - i.e. there are sufficient water droplets in the atmosphere for the phenomena to take place.
Since the position of a rainbow is relative to the position of the observer, we know that a rainbow's location will differ for people at various positions and locations- not everyone views a rainbow from the same position and location!
Another reason why people do not see the exactly identical rainbows is because rainbows occur when light is reflected to the eye precisely at 42 degrees from the original ray. This is where the difference lies. The position of our eyes is relative to, for example, physical factors such as height, body structure etc. All our eyes are not located at the same position. Hence, given that this angle of reflection is dependent on the position of the eye (angle of reflection measured from position of eye), no one can actually see the same rainbow.