float to DWORD conversion bug

Thread view:

Enable EMail Alerts

Start New Thread

Thread rating:

Gabest - 26 Sep 2004 22:07 GMT

Running this piece of code while having the sse optimization turned on
(vcnet2003), something really strange happens I cannot explain. Without sse
it is giving me the right results.

float f = 0.8;
f *= UINT_MAX;
DWORD dw = (DWORD)f;

Basically the result in dw should be 0xcccccccc (or 0xcccccd00 because of
float's inaccuracy), but instead I get 0x80000000 for anything above f =
0.5!

Here is the same code with the assembly instructions inlined copied from the
debugger's view. According to the watch window, f still holds the right
value after f *= UINT_MAX, and only becomes wrong after the third line.

float f = 0.8;

movss xmm0,dword ptr [__real@3f4ccccd (102261C4h)]
movss dword ptr [f],xmm0

f *= UINT_MAX;

movss xmm0,dword ptr [f]
mulss xmm0,dword ptr [__real@4f800000 (102260ECh)]
movss dword ptr [f],xmm0

DWORD dw = (DWORD)f;

fld dword ptr [f]
fnstcw word ptr [ebp-10Ah]
movzx eax,word ptr [ebp-10Ah]
or ah,0Ch
mov dword ptr [ebp-110h],eax
fldcw word ptr [ebp-110h]
fistp dword ptr [ebp-114h]
fldcw word ptr [ebp-10Ah]
mov eax,dword ptr [ebp-114h]
mov dword ptr [dw],eax

Reply to this Message

Gabest - 27 Sep 2004 19:56 GMT

Hm, no one has commented on this yet?

Anyway, I just noticed the problem is not there with the sse opt, but only
with sse2. The sse code seems to call __ftol2 which still works. So as it
looks, the current sse2 option in vc2k3 is quite useless and _dangerous_ if
someone wants to use floating point calculations in his program too.

DWORD dw = (DWORD)f;

sse:

fld dword ptr [f]
call @ILT+1375(__ftol2) (411564h)
mov dword ptr [dw],eax

sse2

fld dword ptr [f]
fnstcw word ptr [ebp-0DAh]
movzx eax,word ptr [ebp-0DAh]
or ah,0Ch
mov dword ptr [ebp-0E0h],eax
fldcw word ptr [ebp-0E0h]
fistp dword ptr [ebp-0E4h]
fldcw word ptr [ebp-0DAh]
mov eax,dword ptr [ebp-0E4h]
mov dword ptr [dw],eax

Reply to this Message

float to DWORD conversion bug

Free Magazines