Skip to content

Wrong printing of Fixed numbers #307

@PatrickHaecker

Description

@PatrickHaecker

Either I have totally misunderstood the documentation or printing of Fixed numbers is wrong.

julia> Fixed{Int8,2}(0.25)
0.2Q5f2

julia> Fixed{Int8,3}(0.125)
0.12Q4f3

Looking into FixedPointNumbers.jl/show leads to

digits=ceil(Int, f * log10_2)

which is wrong, as for 2^-f the number of decimal fraction digits scales linearly and not logarithmic with f's number of binary fraction digits, as you can see here

julia> 2.0.^(-1:-1:-10)
10-element Vector{Float64}:
 0.5
 0.25
 0.125
 0.0625
 0.03125
 0.015625
 0.0078125
 0.00390625
 0.001953125
 0.0009765625

Consequently this can be fixed by using digits=f.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions