A coated article is provided with a low-emissivity (low-E) layer stack for use in a window unit or the like. The layer stack, or coating, may permit the coated article to achieve one or more of a low solar factor (SF) value, a high selectivity (Tvis / SF) value, substantially neutral color at normal and / or off-axis viewing angles, and / or low emissivity. When high selectivity values are achieved, there is provided a high ratio of visible transmission to SF, which is a desirable feature in certain example instances. In certain example embodiments, asymmetric IR reflecting layer thicknesses may be used (e.g., bottom IR reflecting layer thicker than the top IR reflecting layer), and / or different contact layer for the top / bottom IR reflecting layers may be used to help achieve a combination of low SF values, high selectivity, and good coloration at normal and / or off-axis viewing angles.