Questions tagged [precision]
For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].
precision
4,593
questions
0
votes
0
answers
33
views
how can i get the value that is displayed when printing a np.float32 value as float? (or int)
I am loading an openEXR files that I have rendered int values into.
Of course they are stored as float32 internally, which can/will cause issues.
But here i am a bit stumped.
When I load the values, ...
- 121
1
vote
1
answer
38
views
Detect precision loss in numpy floats
If the difference between two numbers is too large, the smallest number will be "lost". Does this raise any flag that I can check?
In float32, the significant has 24 binary digits, therefore ...
- 535
-2
votes
0
answers
15
views
Continuous integration: floating point differences between remote and local
I run continuous integration tests on a (Python) scientific code base I develop.
Some tests are stability tests: I ran some function for a given set of parameters, checked its output manually (for ...
- 383
0
votes
1
answer
24
views
mpmath and mathmatica discrepancy in output
edited because my main issue lies with incorrect results from mp.math.
I need to eventually do very precise calculations with complex numbers, which is why I am trying to use the mp module from mp....
- 3
0
votes
1
answer
25
views
What is the hard and fast rule for GMP mpf precision to number of significant digits
GMP currently works out of the box with a precision of 64 precision bits (I think that's what mp_bitcnt_t is called).
For me this works up to numbers of value up to around 1x10e-22 I think (where I'm ...
- 121
-1
votes
0
answers
16
views
Precision and Scale Question in Floating-Point Column [duplicate]
I'm a little confused by the very last example. Here's the excerpt:
When using a floating-point type, you can specify a precision (the total number of allowable digits both to the left and to the ...
0
votes
0
answers
24
views
Integration using BigFloat in Julia
I have a problem with understanding how calculation with BigFloat works in Julia. I want to calculate an integral using the DoubleExponentialFormulas QuadDE function.
I thought this part would work:
...
0
votes
1
answer
79
views
How to make python use more than 15 decimals in calculations?
In this question I made a claim that this construction of pentagons will always converge.
I drew the first 50 iteration using Geogebra here. However, zooming in on the 50th iteration resulted in ...
- 95
-1
votes
1
answer
179
views
Unexpected result in floating point output
The following code:
#include <iostream>
#include <ios>
int main() {
auto mod = 1e9+7;
std::cout << mod << '\n';
std::cout << std::fixed << mod << ...
- 21
1
vote
1
answer
66
views
Is it correct and consistent to check two vertex points for equality?
I am writing a Typescript game where there are vertexes drawn to a canvas, and some edges coming from them. I want to determine if some of edges from different vertexes intersect. For this, I am using ...
- 11.7k
2
votes
2
answers
107
views
Polars truncates decimals
I'm trying to truncate floating point numbers in my DataFrame to a desired number of decimal places. I've found that this can be done using Pandas and NumPy here, but I've also seen that it might be ...
- 437
0
votes
1
answer
26
views
Oracle looses float precision when using a trigger
I use a trigger to fill a SDO_POINT_TYPE, or SDO_GEOMETRY column from two other columns of the same table.
For some reason, the trigger changes the floating values precision when compared with direct ...
- 1,775
0
votes
0
answers
37
views
Increasing Precision of Eigenvectors (Matrix with Large Eigenvalue)
I am currently using scipy.linalg.eigh to find the eigenvectors/eigenvalues of a symmetric matrix. I found that as I increase the size of the matrix, the eigenvectors started losing precision. ...
- 1
0
votes
0
answers
60
views
Trouble with double-precision floats while replicating of one Software's behavior in Python
I struggle with a somewhat a problem for my brain and would appreciate your thoughts and help to resolve this.
I have a software (very very old.. no versions available anymore on the internet) I would ...
0
votes
0
answers
33
views
Precision issue in R: Why does dividing by 0.00001 in the operation 1/0.00001 cause problems, while other magnitudes do not? [duplicate]
When trying to run a simulation experiment with the following code, I found that the results returned had issues. Why is this happening?
> length(rnorm(100000))
[1] 100000
> length(rnorm(1/0....
- 1