Sensor API Reference¶

This section documents sensor simulation and image signal processing.

Sensor Classes¶

RGBSensor¶

class deeplens.sensor.RGBSensor(sensor_file)¶

RGB sensor with Bayer color filter array driven by a configuration file.

Parameters:: sensor_file – Path to a JSON file describing sensor parameters

The constructor reads key properties from sensor_file (bit depth, black level, Bayer pattern, noise statistics, optional white balance/color matrix/gamma, and optional spectral responses). Internally it builds an deeplens.sensor.isp.InvertibleISP.

Key attributes:

size: Physical sensor size (W, H) in mm
res: Sensor resolution (W, H) in pixels
bit: ADC bit depth
black_level: Black level offset in DN
bayer_pattern: Bayer pattern string (e.g., "rggb")
iso_base, read_noise_std, shot_noise_std_alpha, shot_noise_std_beta
isp: deeplens.sensor.isp.InvertibleISP

Selected methods:

forward(img_nbit, iso)¶

Simulate sensor noise and run ISP.

Parameters:

img_nbit – RAW RGB or Bayer-space tensor [B, 3, H, W] in n-bit DN
iso – ISO value(s) [B]

Returns:

RGB image [B, 3, H, W] in [0, 1]

unprocess(image, in_type='rgb')¶: Convert RGB to unbalanced RAW RGB (inverse gamma/CCM/AWB).

linrgb2bayer(img_linrgb)¶: Convert linear RGB [0, 1] to n-bit Bayer [~black_level, 2**bit - 1].

process2rgb(image, in_type='rggb')¶: Process Bayer or RGGB to RGB using the internal ISP.

bayer2rggb(bayer_nbit) / rggb2bayer(rggb)¶: Pack/unpack between Bayer [B,1,H,W] and RGGB [B,4,H/2,W/2].

MonoSensor¶

class deeplens.sensor.MonoSensor(resolution=(2048, 2048), pixel_size=3.45e-3, bit_depth=14, qe=0.7, device='cuda')¶

Monochrome sensor without color filter.

Parameters:

resolution – Sensor resolution
pixel_size – Pixel size in mm
bit_depth – ADC bit depth
qe – Quantum efficiency
device – Device

Methods:

capture(irradiance, exposure_time=0.01, iso=100)¶

Capture monochrome image.

Parameters:

irradiance – Irradiance [W/m²] [H, W]
exposure_time – Exposure time
iso – ISO

Returns:

Sensor data [H, W]

EventSensor¶

class deeplens.sensor.EventSensor(resolution=(640, 480), pixel_size=18.5e-3, threshold=0.1, refractory_period=1e-3, device='cuda')¶

Event-based (DVS) sensor.

Parameters:

resolution – Sensor resolution
pixel_size – Pixel size in mm
threshold – Contrast threshold for events
refractory_period – Refractory period in seconds
device – Device

Methods:

capture_events(frame_sequence, timestamps)¶

Generate events from frame sequence.

Parameters:

frame_sequence – Sequence of frames [T, H, W]
timestamps – Time for each frame [T]

Returns:

Events (x, y, t, p) where p is polarity

ISP Pipeline¶

ISP¶

class deeplens.sensor.ISP(demosaic_method='bilinear', white_balance=True, color_correction=True, gamma_correction=True, denoise=False, sharpen=False, device='cuda')¶

Complete Image Signal Processing pipeline.

Parameters:

demosaic_method – Demosaicing algorithm
white_balance – Enable white balance
color_correction – Enable color correction
gamma_correction – Enable gamma correction
denoise – Enable denoising
sharpen – Enable sharpening
device – Device

Methods:

forward(raw_image)¶

Process raw sensor data.

Parameters:: raw_image – Raw Bayer image [H, W]
Returns:: Processed RGB image [3, H, W]

set_ccm(matrix)¶

Set color correction matrix.

Parameters:: matrix – 3x3 color correction matrix

set_wb_gains(gains)¶

Set white balance gains.

Parameters:: gains – [R, G, B] gains

ISP Modules¶

BlackLevel¶

class deeplens.sensor.isp_modules.BlackLevel(level=64, device='cuda')¶

Black level correction.

Parameters:

level – Black level offset
device – Device

forward(raw)¶

Apply black level correction.

Parameters:: raw – Raw sensor data
Returns:: Corrected data

LensShadingCorrection¶

class deeplens.sensor.isp_modules.LensShadingCorrection(resolution, center=None, falloff=0.3, device='cuda')¶

Lens shading correction for vignetting.

Parameters:

resolution – Image resolution (W, H)
center – Optical center [cx, cy] (default: image center)
falloff – Vignetting falloff factor
device – Device

forward(raw)¶

Apply lens shading correction.

Parameters:: raw – Raw image
Returns:: Corrected image

DeadPixelCorrection¶

class deeplens.sensor.isp_modules.DeadPixelCorrection(threshold=0.1, method='median', device='cuda')¶

Dead and hot pixel correction.

Parameters:

threshold – Detection threshold
method – Correction method (‘median’, ‘mean’)
device – Device

forward(raw)¶

Correct dead pixels.

Parameters:: raw – Raw image
Returns:: Corrected image

WhiteBalance¶

class deeplens.sensor.isp_modules.WhiteBalance(method='gray_world', gains=None, device='cuda')¶

White balance correction.

Parameters:

method – ‘gray_world’, ‘white_patch’, or ‘manual’
gains – Manual gains [R, G, B] (for manual mode)
device – Device

forward(raw)¶

Apply white balance.

Parameters:: raw – Raw Bayer image
Returns:: Balanced image

estimate_gains(raw)¶

Estimate white balance gains.

Parameters:: raw – Raw image
Returns:: Estimated [R, G, B] gains

Demosaic¶

class deeplens.sensor.isp_modules.Demosaic(method='bilinear', device='cuda')¶

Bayer demosaicing.

Parameters:

method – ‘bilinear’, ‘malvar’, ‘menon’, or ‘ahd’
device – Device

forward(bayer)¶

Demosaic Bayer pattern to RGB.

Parameters:: bayer – Bayer image [H, W]
Returns:: RGB image [3, H, W]

Available Methods:

bilinear: Fast bilinear interpolation
malvar: Edge-aware interpolation
menon: High-quality edge-directed
ahd: Adaptive homogeneity-directed

ColorMatrix¶

class deeplens.sensor.isp_modules.ColorMatrix(matrix=None, device='cuda')¶

Color correction matrix.

Parameters:

matrix – 3x3 correction matrix (default: identity)
device – Device

forward(rgb)¶

Apply color correction.

Parameters:: rgb – RGB image [3, H, W]
Returns:: Corrected RGB [3, H, W]

GammaCorrection¶

class deeplens.sensor.isp_modules.GammaCorrection(gamma=2.2, method='power', device='cuda')¶

Gamma correction for display.

Parameters:

gamma – Gamma value
method – ‘power’, ‘srgb’, or ‘log’
device – Device

forward(linear_rgb)¶

Apply gamma correction.

Parameters:: linear_rgb – Linear RGB [3, H, W]
Returns:: Gamma-corrected RGB [3, H, W]

Denoise¶

class deeplens.sensor.isp_modules.Denoise(method='bilateral', strength=0.5, device='cuda')¶

Image denoising.

Parameters:

method – ‘bilateral’, ‘nlm’, or ‘bm3d’
strength – Denoising strength
device – Device

forward(rgb)¶

Denoise image.

Parameters:: rgb – RGB image [3, H, W]
Returns:: Denoised image [3, H, W]

ColorSpace¶

class deeplens.sensor.isp_modules.ColorSpace(device='cuda')¶

Color space conversions.

Parameters:: device – Device

rgb_to_yuv(rgb)¶

RGB to YUV conversion.

Parameters:: rgb – RGB image [3, H, W]
Returns:: YUV image [3, H, W]

yuv_to_rgb(yuv)¶

YUV to RGB conversion.

Parameters:: yuv – YUV image [3, H, W]
Returns:: RGB image [3, H, W]

rgb_to_hsv(rgb)¶

RGB to HSV conversion.

Parameters:: rgb – RGB image [3, H, W]
Returns:: HSV image [3, H, W]

srgb_to_linear(srgb)¶

sRGB to linear RGB.

Parameters:: srgb – sRGB image [3, H, W]
Returns:: Linear RGB [3, H, W]

linear_to_srgb(linear)¶

Linear RGB to sRGB.

Parameters:: linear – Linear RGB [3, H, W]
Returns:: sRGB image [3, H, W]

Sharpen¶

class deeplens.sensor.isp_modules.Sharpen(strength=0.3, radius=1.0, device='cuda')¶

Image sharpening.

Parameters:

strength – Sharpening strength
radius – Sharpening radius
device – Device

forward(rgb)¶

Sharpen image.

Parameters:: rgb – RGB image [3, H, W]
Returns:: Sharpened image [3, H, W]

AntiAliasing¶

class deeplens.sensor.isp_modules.AntiAliasing(sigma=0.5, kernel_size=5, device='cuda')¶

Anti-aliasing filter.

Parameters:

sigma – Gaussian blur sigma
kernel_size – Filter kernel size
device – Device

forward(image)¶

Apply anti-aliasing.

Parameters:: image – Input image
Returns:: Filtered image

Examples¶

Basic Sensor Usage¶

from deeplens.sensor import RGBSensor

# Create sensor from configuration file
sensor = RGBSensor(sensor_file='./datasets/sensors/imx586.json')

# Process raw image with noise and ISP
# img_nbit: raw RGB tensor [B, 3, H, W] in n-bit DN range
img_rgb = sensor.forward(img_nbit, iso=100)

Complete ISP Pipeline¶

from deeplens.sensor import RGBSensor

# Create sensor with built-in ISP from config file
sensor = RGBSensor(sensor_file='./datasets/sensors/imx586.json')

# The sensor includes an InvertibleISP pipeline
# Forward: raw -> noise -> ISP -> RGB
img_rgb = sensor.forward(img_nbit, iso=100)

# Reverse: RGB -> unprocess -> linear RGB -> raw
img_raw = sensor.unprocess(img_rgb, in_type='rgb')
bayer = sensor.linrgb2bayer(img_raw)

Custom ISP Pipeline¶

from deeplens.sensor.isp_modules import *
import torch.nn as nn

class MyISP(nn.Module):
    def __init__(self):
        super().__init__()
        self.black_level = BlackLevel(level=64)
        self.wb = WhiteBalance(method='gray_world')
        self.demosaic = Demosaic(method='malvar')
        self.ccm = ColorMatrix()
        self.gamma = GammaCorrection(gamma=2.2)

    def forward(self, raw):
        x = self.black_level(raw)
        x = self.wb(x)
        x = self.demosaic(x)
        x = self.ccm(x)
        x = self.gamma(x)
        return x

# Use custom ISP
my_isp = MyISP()
rgb = my_isp(raw_image)

Camera with Sensor¶

from deeplens import Camera, GeoLens
from deeplens.sensor import RGBSensor, ISP

# Create components
lens = GeoLens(filename='lens.json')
sensor = RGBSensor(resolution=(1920, 1080))
isp = ISP()

# Create camera
camera = Camera(lens=lens, sensor=sensor, isp=isp)

# Capture image
img = camera.capture(scene, depth=1000, exposure_time=0.01)

Noise Simulation¶

from deeplens.sensor import RGBSensor

# Create sensor from config file
# Noise parameters are specified in the config JSON:
# - iso_base, read_noise_std, shot_noise_std_alpha, shot_noise_std_beta
sensor = RGBSensor(sensor_file='./datasets/sensors/imx586.json')

# Forward pass applies noise based on ISO setting
# Higher ISO = more noise amplification
img_noisy_low_iso = sensor.forward(img_nbit, iso=100)
img_noisy_high_iso = sensor.forward(img_nbit, iso=3200)

White Balance Estimation¶

from deeplens.sensor.isp_modules import WhiteBalance

wb = WhiteBalance(method='gray_world')

# Estimate gains from image
gains = wb.estimate_gains(raw_image)
print(f"WB gains: R={gains[0]:.2f}, G={gains[1]:.2f}, B={gains[2]:.2f}")

# Apply white balance
balanced = wb(raw_image)

DeepLens

Navigation

Related Topics

Sensor API Reference¶

Sensor Classes¶

RGBSensor¶

MonoSensor¶

EventSensor¶

ISP Pipeline¶

ISP¶

ISP Modules¶

BlackLevel¶

LensShadingCorrection¶

DeadPixelCorrection¶

WhiteBalance¶

Demosaic¶

ColorMatrix¶

GammaCorrection¶

Denoise¶

ColorSpace¶

Sharpen¶

AntiAliasing¶

Examples¶

Basic Sensor Usage¶

Complete ISP Pipeline¶

Custom ISP Pipeline¶

Camera with Sensor¶

Noise Simulation¶

White Balance Estimation¶

See Also¶