Questions tagged [floating-point]
Floating-point numbers are approximations of real numbers that can represent larger ranges than integers but use the same amount of memory, at the cost of lower precision. If your question is about small arithmetic errors (e.g. why does 0.1 + 0.2 equal 0.300000001?) or decimal conversion errors, please read the tag page before posting.
floating-point
15,314
questions
0
votes
1
answer
62
views
Lossy conversion between long double and double
I was cought by suprise that the following code returns false for gcc 13 and clang 18. Why does this happen? Isn't the number 8.1 representable in both formats?
#include <iostream>
#include <...
- 948
1
vote
1
answer
35
views
Detect precision loss in numpy floats
If the difference between two numbers is too large, the smallest number will be "lost". Does this raise any flag that I can check?
In float32, the significant has 24 binary digits, therefore ...
- 535
1
vote
0
answers
20
views
Inconsistent Texture Rendering in Raycasting Due to Intersection Calculations
I'm working on a raycasting engine where I'm encountering an issue with inconsistent texture rendering. When rays hit consecutive walls of the same texture (e.g., 1 1 1 in the map), some rays ...
- 73
-3
votes
0
answers
53
views
Why is it showing this output instead of my assigned variable? [duplicate]
public class pr1 {
public static void main(String[] args){
double f = 123.456f;
System.out.println(f);
//output is 123.45600128173828
I have used the double data type ...
- 11
0
votes
3
answers
250
views
Double floating point operations using four IEEE rounding modes implemented in terms of round to nearest ties to even
I am looking to implement the RandomX Proof of Work hash function in JavaScript + WebAssembly. The README explains how this isn't possible:
Web mining is infeasible due to the large memory ...
- 1
1
vote
2
answers
78
views
Why is Float truncating a 10-digit number?
I'm aware of some issue with floating point precision, but I don't understand this one:
When storing 9138497.986 as a Float(), it appears to be getting truncated to "9138497". Can someone ...
- 8,233
3
votes
1
answer
114
views
Difference between integer division vs floating point division + int conversion
Assuming x ≥ 0 and y > 0 are both 64-bit unsigned integers would there be any correctness difference between below implementation and x // y?
def int_divide(x, y):
return int(x / float(y))
- 1,567
1
vote
2
answers
104
views
Incorrect calculation of a math expression using sin and cos in C?
Why does this simple program
#include <stdio.h>
#include <math.h>
int main()
{
int res = (int)(-1 * sin(M_PI/2) + 1 * cos(M_PI/2));
printf("-1 * %f + 1 * %f = %d\n", sin(...
- 83
1
vote
2
answers
67
views
How to elegantly read/write a struct field as little endian float?
I've written some code that intercepts data packets and modifies them. 4 bytes in the data packets represent a float value, but in reverse byte order (little-endian). And that creates a lot of issues ...
- 15.2k
2
votes
1
answer
75
views
How does rounding works in float multiplication?
The exact value of float 0.7f is 0.69999...
so I thought the result of 0.7f * 100f would be something below 70, like 69.99999...
But the result is exact 70.
Does float multiplication involve such ...
- 23
1
vote
2
answers
53
views
Why floating point of a tensors still fluctuates even after using set_printoptions(precision=1)
I followed a tutorial that shows how to do tensors operations in a correct way. They said that usually the operations between tensors are done manually via loops of iterating through the tensor array. ...
- 13
-2
votes
1
answer
83
views
Why do float decimals reappear even after they are truncated and cast as String? [duplicate]
I need to store a float with n decimals in a variable but it seems too complicated.
If I run this code:
float testFloat = -18.479999542236328f;
System.out.println(getFloatAsStringIWithNDecimals(...
- 6,309
-1
votes
0
answers
16
views
Precision and Scale Question in Floating-Point Column [duplicate]
I'm a little confused by the very last example. Here's the excerpt:
When using a floating-point type, you can specify a precision (the total number of allowable digits both to the left and to the ...
-3
votes
1
answer
59
views
Why different output of numbers?
In this program, when I input 3 numbers 13, 179, 0 I get 2 different outputs from 2 equivalent programs, first - 202 26, second - 202 27. Why?
P = int(input())
X = int(input())
Y = int(input())
# ...
0
votes
0
answers
22
views
Is it some kind of numerical instability for this convolution product (C language)?
I am trying to program a convolution product function in C language.
I am generating sine waves with the following function:
int generate_sin_wave(signal_t *signal, sin_wave_config_t *wave_config, ...
- 214