# Camera–IMU Time Offset Guide

English | [中文](README_TIMESHIFT.md)

This document explains the origin of the camera–IMU time offset reported by Kalibr, why the calibrated value changes with exposure duration, and how the result should be applied at runtime.

## 1. Kalibr time-shift convention

Kalibr writes one value for each camera to `dataset-camchain-imucam.yaml`:

```yaml
timeshift_cam_imu: 0.0027153517330281273
```

The corresponding time relation is:

```text
t_imu = t_cam_header + timeshift_cam_imu
```

- `t_cam_header` is the image message `header.stamp` stored in the rosbag.
- `t_imu` is the IMU time corresponding to the image observation.
- A positive value means that the time shift must be added to the camera timestamp to align it with IMU time.
- `0.002715 s` equals `2.715 ms`.

This value is not simply a wire or transport delay. It combines the fixed clock offset, the position of the camera timestamp within the exposure interval, and other fixed sensor or driver delays.

## 2. Main sources of time offset

### 2.1 Different camera and IMU clock domains

If the camera and IMU do not share a hardware clock, their clocks may have different epochs and frequencies. A one-time software synchronization can remove part of the initial offset, but a fixed residual offset and time-dependent drift may remain.

### 2.2 The image timestamp may not represent the exposure midpoint

An image is acquired over a finite interval. A driver may assign `header.stamp` at any of the following points:

| Timestamp position | Effective image observation time relative to `header.stamp` |
|---|---:|
| Exposure start | `+ exposure / 2` |
| Exposure midpoint | `0` |
| Exposure end | `- exposure / 2` |

Kalibr estimates the observation time from the motion of the calibration target in the images. It therefore absorbs this timestamp-position difference into `timeshift_cam_imu`.

### 2.3 Exposure-duration changes

If the timestamp represents exposure start and the effective image observation occurs at exposure midpoint, then:

```text
timeshift_kalibr(E) = timeshift_fixed + E / 2
```

where:

- `E` is the exposure duration;
- `timeshift_fixed` is the fixed camera–IMU system offset after removing the exposure contribution.

Increasing exposure by `1 ms` should therefore increase the Kalibr result by approximately `0.5 ms`. This does not mean that the spatial extrinsic calibration has changed. It means that the effective image observation time has moved.

### 2.4 Driver, ISP, compression, and message-queue delays

If an image timestamp is based on receive time rather than the hardware acquisition time, ISP processing, JPEG compression, USB or network transport, and ROS queue delays may also enter the calibrated offset. These delays may vary with system load and appear as jitter rather than a constant offset.

The IMU path may also introduce digital-filter group delay, packet batching, or differences in the timestamp position chosen by the driver.

### 2.5 Differences among multiple cameras

Even when four cameras share a trigger, small timestamp differences can be introduced by driver timestamp order, queues, or independent clocks. The current system keeps one independent time-shift parameter per camera and requires the final four-camera spread to remain within `100 us`. It does not force all four parameters to be identical in the optimizer.

## 3. Measurements from the e1 and e2 datasets

Both calibrations used the same fixed EUCM camera intrinsics and the same IMU parameters:

| Dataset | Exposure | cam0 | cam1 | cam2 | cam3 | Four-camera mean | Four-camera range |
|---|---:|---:|---:|---:|---:|---:|---:|
| e1 | 1000 us | 2.067305 ms | 2.078715 ms | 2.102514 ms | 2.091259 ms | 2.084948 ms | 35.209 us |
| e2 | 2000 us | 2.715352 ms | 2.665352 ms | 2.665352 ms | 2.665659 ms | 2.677929 ms | 50.000 us |

The difference between the two raw means is:

```text
2.677929 ms - 2.084948 ms = 0.592980 ms
```

Exposure increased from `1 ms` to `2 ms`. If the timestamp is assigned at exposure start, the theoretical change in the exposure midpoint is:

```text
(2 ms - 1 ms) / 2 = 0.500 ms
```

The measurement is broadly consistent with this model. The remaining difference of approximately `92.98 us` can be attributed to statistical variation between the two recordings, motion-trajectory differences, image-detection errors, and system timing jitter.

## 4. Fixed offset after exposure compensation

The two current datasets provide initial support for the exposure-start timestamp model. Under this assumption, use:

```text
timeshift_fixed = timeshift_kalibr - exposure / 2
```

This gives:

| Dataset | Raw mean | Half exposure removed | Estimated fixed offset |
|---|---:|---:|---:|
| e1 | 2.084948 ms | 0.500 ms | 1.584948 ms |
| e2 | 2.677929 ms | 1.000 ms | 1.677929 ms |

The simple mean of these two estimates is:

```text
timeshift_fixed ≈ 1.631 ms
```

This can be used as a reference fixed offset for the current device. Because only two exposure settings have been measured, more exposure settings and repeated datasets should be tested before choosing a production value.

> Only subtract half the exposure after confirming that `header.stamp` represents exposure start. If the driver already timestamps the exposure midpoint, subtracting half the exposure again would apply the compensation twice.

## 5. Runtime usage

### Option A: keep the raw exposure-start timestamp

If the image `header.stamp` still represents exposure start and `exposure_us` is available for every frame:

```text
t_image_effective = t_cam_header + exposure_us * 1e-6 / 2
t_imu_aligned     = t_image_effective + timeshift_fixed
```

Using the current reference value:

```text
t_imu_aligned = t_cam_header + exposure_us * 1e-6 / 2 + 0.001631
```

### Option B: publish exposure-midpoint timestamps from the driver

The preferred solution is to correct the image timestamp to exposure midpoint in the driver:

```text
t_cam_mid = t_cam_exposure_start + exposure_us * 1e-6 / 2
```

Runtime alignment then only needs the fixed offset:

```text
t_imu_aligned = t_cam_mid + timeshift_fixed
```

### Option C: use one raw Kalibr result directly

If the runtime exposure exactly matches the exposure used during calibration, the raw value from the corresponding YAML can be used directly:

```text
t_imu_aligned = t_cam_header + timeshift_cam_imu
```

This method cannot be reused directly across exposure settings. When exposure changes, either recalibrate or use Option A/B.

## 6. Confirming the timestamp position

Record datasets at no fewer than four exposure settings using similarly rich rotational motion. Calibrate each dataset and fit:

```text
timeshift_kalibr(E) = a + k * E
```

- `k ≈ +0.5`: timestamp is close to exposure start;
- `k ≈ 0`: timestamp is close to exposure midpoint;
- `k ≈ -0.5`: timestamp is close to exposure end;
- an unstable slope: the system may be using software receive timestamps, suffering from queue jitter, or lacking sufficient motion coverage.

The intercept `a` estimates the fixed system offset after exposure normalization.

## 7. Acceptance recommendations

Check at least the following for every camera–IMU calibration:

1. The maximum spread among the four `timeshift_cam_imu` values is no more than `100 us`.
2. No time-shift parameter is saturated at an optimization boundary.
3. After applying the correct exposure term, the fixed offset remains stable across exposure settings.
4. Camera–IMU rotation and translation remain consistent across repeated datasets.
5. Reprojection, gyroscope, and accelerometer residuals do not degrade significantly.
6. All four image streams in a rosbag use the same exposure setting and publish an accurate `exposure_us` value.

For the current e1/e2 recordings, the spatial-extrinsic repeatability is better than approximately `0.1 deg / 0.9 mm`, while the four-camera time-shift ranges are `35.2 us` and `50.0 us`. The overall results are therefore valid. The exposure-dependent time shift should be handled using the formulas above and should not be interpreted as a change in the physical camera–IMU mounting extrinsics.