Real Datatype integer accuracy

M Donaldson 2 years ago in IQANdesign updated by Ulrik Zakariasson (Software development) 2 years ago 3

How many bits are part of the mantissa for the real datatype? We are bring in a rather large value(32-bit) that we use to generate PIN codes. Unfortunately we are bringing this in with a JPIN which is a real datatype. I've noticed in simulation when I enter a certain value it seems to get rounded to another. See below

Typed Value End Value
20031540 20031540
20031541 20031540
20031542 20031542
20031543 20031544
20031544 20031544
20031545 20031544
20031546 20031546
20031547 20031548
20031548 20031548
20031549 20031548

 I'm just wondering what the top end is for Integer accuracy using a real datatype.


We are using single precision floating point values (32-bit floating point). This means that you will get 7 significant digits. For numbers with more than 7 significant digits there will be a loss of precision.

You can connect a GPIN channel to the JFIN to bring in a 32-bit integer value without scaling and offset. This will keep the original integer value. Note however that unsigned 32-bit is not supported and will converted to signed integer meaning that the maximum value will be 2147483647.