- Software
- IQANdesign
-
Questions
0
Answered
Real Datatype integer accuracy
M Donaldson 7 years ago
in IQANdesign
•
updated by Ulrik Zakariasson (Software development) 7 years ago •
3
How many bits are part of the mantissa for the real datatype? We are bring in a rather large value(32-bit) that we use to generate PIN codes. Unfortunately we are bringing this in with a JPIN which is a real datatype. I've noticed in simulation when I enter a certain value it seems to get rounded to another. See below
| Typed Value | End Value |
| 20031540 | 20031540 |
| 20031541 | 20031540 |
| 20031542 | 20031542 |
| 20031543 | 20031544 |
| 20031544 | 20031544 |
| 20031545 | 20031544 |
| 20031546 | 20031546 |
| 20031547 | 20031548 |
| 20031548 | 20031548 |
| 20031549 | 20031548 |
I'm just wondering what the top end is for Integer accuracy using a real datatype.
Thanks,
Customer support service by UserEcho
We are using single precision floating point values (32-bit floating point). This means that you will get 7 significant digits. For numbers with more than 7 significant digits there will be a loss of precision.
You can connect a GPIN channel to the JFIN to bring in a 32-bit integer value without scaling and offset. This will keep the original integer value. Note however that unsigned 32-bit is not supported and will converted to signed integer meaning that the maximum value will be 2147483647.
Thank You