Sensor Curve Calibration: When the Reference Position Also Has Error

Measurement purpose

Select a forward calibration fit consistent with material uncertainty in both coordinates.

Specimens and conditions

Reference state

Known physical position definition and stable repeated states.

Sensor operation

Defined travel direction, contact state, temperature and electrical stimulus.

Equipment and records required

• Position reference: Characterized random and systematic errors with units and replicate structure.
• Resistance acquisition: Controlled loading and measurement uncertainty connected to the paired observations.

Method sequence

1. Characterize

   Separate coordinate errors from between-position span.

   Record: Error model and variance evidence.

2. Fit

   Apply and independently check the declared fitting objective.

   Record: Coefficients, unit convention and software challenge.

3. Validate

   Use independent positions to check predictions.

   Record: Confirmation residuals and unresolved model limitations.

Decision and uncertainty

Select a model supported by coordinate-error evidence rather than visual line agreement.

Variance-ratio error, shared references and systematic scale effects remain distinct.

Calibration-method and sensor-system owners.

Traceable outputs

Separate true position from its observed indication

For a resistive position sensor, let the underlying position be X and the measured resistance be Y. The fixture reports x, an imperfect observation of X, while the resistance instrument reports y. A forward linear model relates the underlying values through Y = a + bX over a specifically justified interval. Measurement errors then affect both displayed coordinates.

This is different from predicting position from a completed calibration or selecting withheld confirmation points. The immediate question is how to estimate the forward relation without assigning fixture-coordinate noise to the printed track. Identify whether the reference measures the actual contact position, shaft angle or another mechanical point. A perfect readout of the wrong physical coordinate does not become a valid reference through a more sophisticated regression.

Identify when the ordinary least-squares assumption matters

Ordinary least squares minimizes squared vertical response residuals while treating the supplied x values as fixed for the fit. If random reference-coordinate error is important, part of the horizontal spread is measurement error rather than true span. Under a simple independent classical-error model, fitting as though that spread were entirely real can pull the estimated slope toward zero. This is not an inevitable behavior for every possible calibration design or error model.

Compare repeatability at fixed reference states with the spread between the distinct calibration positions. Also inspect systematic scale error, backlash and temperature effects. Repeated readings can characterize random indication noise but will not reveal an uncorrected reference scale bias by themselves. Do not select ordinary least squares simply because the position display has more digits than the resistance display.

Establish the variance ratio before running Deming regression

For the simple constant-variance model used here, define lambda as the variance of resistance measurement error divided by the variance of position measurement error. If y is recorded in kilohms and x in degrees, lambda has units of kilohms squared per degree squared. A value of one therefore means numerical equality in those chosen units, not equal physical uncertainty in a universal sense.

Estimate relevant repeatability from suitably repeated measurements at stable underlying states, or use an independently justified error model. If the fitted points are means of different numbers of replicates, their variances differ from the variances of individual readings. A single constant ratio may then be inadequate. State explicitly whether the software expects lambda or its reciprocal and verify the result using a known calculation before applying the routine to customer data.

Fit the latent coordinates without inventing extra observations

A two-coordinate fit can minimize the sum of squared horizontal corrections divided by position-error variance plus squared vertical residuals divided by response-error variance. The unknown underlying coordinates are fitted along with the line. For independent, zero-mean errors with constant known variance ratio, this leads to the straight-line Deming solution. The corrected coordinates are model estimates, not additional physical measurements.

For centered sums Sxx, Syy and Sxy, the positive-covariance slope is given below. The intercept is the mean y minus the slope times mean x. Keep the chosen units throughout. If covariance is zero or nearly zero, the expression can be unstable or inappropriate for a useful calibration. Do not force a finite-looking slope from a data set that does not resolve the intended relationship.

Compare two fitting assumptions on the same summary data

Consider illustrative paired observations with mean position 5 degrees, mean resistance 11 kilohms, Sxx = 10 degrees squared, Syy = 40 kilohms squared and Sxy = 18 degree-kilohms. Ordinary least squares gives a slope of 18/10 = 1.8 kilohms per degree and an intercept of 2 kilohms. These summary values are deliberately chosen to make the fitting distinction visible; they are not a sensor specification.

With an independently assigned lambda of 4 kilohms squared per degree squared, Syy minus lambda Sxx is zero. The Deming square-root term is 72 kilohms squared, giving slope 2 kilohms per degree and intercept 1 kilohm. Both lines pass through the same coordinate means. Their difference reflects the allocated coordinate-error model, not evidence that the second line must be physically correct.

Do not use a single ratio to absorb every calibration defect

If both instruments share a reference or environmental correction, their errors may be correlated. If uncertainty changes with position, constant weighting can distort the contribution of different parts of the travel. Nonlinear track response, directional hysteresis and contact interruptions also need models or stratified observations that represent those mechanisms. A straight line with adjusted weighting does not remove physical nonlinearity.

Treat systematic position scale and resistance gain errors through their calibration evidence. They can shift or rotate the entire observed relation while leaving repeatability excellent. Check residual patterns against position, direction, time and contact history rather than inspecting only a single fit statistic. An apparently improved residual plot is not sufficient reason to replace the original measurement model.

Verify implementation and test the fitted relation independently

Use a reproducible calculation with known units and a documented variance convention to challenge the fitting software. Confirm that changing resistance units rescales the coefficients correctly when lambda is rescaled consistently. Preserve the original observations so a later analyst can reproduce the fit without relying on the routine's adjusted-coordinate output.

Then evaluate the relation on independent positions and repeat operating states that were not used to choose its form. Record prediction differences with the relevant reference uncertainty. A calibration-fit confidence interval is not automatically the uncertainty of a new position inferred from resistance. That inverse decision must include local sensitivity, measurement contributions and any nonunique part of the response separately.

Deliver a fit decision with its measurement assumptions

The handoff should include the forward equation, coordinate definitions, units, replicate structure and the evidence supporting the error allocation. Keep rejected model assumptions and their reasons in the engineering record. If the reference uncertainty is not sufficiently characterized, identify the measurement needed to resolve it rather than selecting an arbitrary equal-error option.

For a custom resistive sensor circuit review, ChipSimple needs the mechanical position definition, electrical readout, working interval and calibration data. The useful output is a defensible relation between position and resistance under those conditions. It does not establish a universal accuracy grade or replace the separate evaluation of contact wear, environmental change and assembly positioning.