Skip to content
Snippets Groups Projects
Commit 30c760c1 authored by Bernd Porr's avatar Bernd Porr Committed by Greg Kroah-Hartman
Browse files

staging: comedi: usbduxsigma: fixing rounding error in the range struct


The AD converter on the usbsigma turns vref (2.5V/2) into the
binary value 0x780000. However the full scale range goes up to
0x800000 so that vref/2 needs to be upscaled by 0x800000/0x780000
which is 1.06bar. In the driver it's been 1.06 without the infinite
6666. I've put the fraction in the code so that the compiler can
calculate the most precise scaling value by itself.

Signed-off-by: default avatarBernd Porr <mail@berndporr.me.uk>
Reviewed-by: default avatarIan Abbott <abbotti@mev.co.uk>
Signed-off-by: default avatarGreg Kroah-Hartman <gregkh@linuxfoundation.org>
parent 0e02dcae
No related branches found
No related tags found
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment