Sensors and ISP

DeepLens provides comprehensive sensor simulation and image signal processing (ISP) pipelines for realistic image capture.

Sensor Types

RGB Sensor

Standard Bayer pattern RGB sensor with color filter array.

from deeplens.sensor import RGBSensor

sensor = RGBSensor(
    resolution=(1920, 1080),    # Width x Height
    pixel_size=4.0e-3,          # Pixel size [mm], 4 micrometers
    bit_depth=12,               # ADC bit depth
    qe=0.6,                     # Quantum efficiency
    dark_current=0.01,          # Dark current [e-/s]
    read_noise=2.0,             # Read noise [e-]
    full_well=10000,            # Full well capacity [e-]
    device='cuda'
)

Bayer Pattern:

The sensor uses a standard RGGB Bayer pattern:

R G R G R G
G B G B G B
R G R G R G
G B G B G B

Mono Sensor

Monochrome sensor without color filter array.

from deeplens.sensor import MonoSensor

sensor = MonoSensor(
    resolution=(2048, 2048),
    pixel_size=3.45e-3,
    bit_depth=14,
    qe=0.7,
    device='cuda'
)

Event Sensor

Event-based sensor (DVS) for high-speed applications.

from deeplens.sensor import EventSensor

sensor = EventSensor(
    resolution=(640, 480),
    pixel_size=18.5e-3,
    threshold=0.1,              # Contrast threshold
    refractory_period=1e-3,     # Refractory period [s]
    device='cuda'
)

Sensor Properties

Key Parameters

Parameter

Description

Typical Range

pixel_size

Physical pixel size

1-20 μm

bit_depth

ADC resolution

8-16 bits

qe

Quantum efficiency

0.3-0.9

dark_current

Dark current noise

0.001-1 e-/s

read_noise

Read noise

1-10 e-

full_well

Well capacity

1k-100k e-

Sensor Characteristics

# Print sensor specifications
print(f"Sensor size: {sensor.sensor_size} mm")
print(f"Resolution: {sensor.resolution} pixels")
print(f"Pixel pitch: {sensor.pixel_size*1000:.2f} μm")
print(f"Sensor diagonal: {sensor.diagonal:.2f} mm")
print(f"Crop factor: {sensor.crop_factor:.2f}")

Image Capture

Basic Capture

# Capture image from optical field
# Input: irradiance on sensor [W/m^2]
irradiance = lens.get_irradiance()  # [H, W, 3] or [H, W]

# Capture with sensor
raw_image = sensor.capture(
    irradiance=irradiance,
    exposure_time=0.01,  # Exposure time [s]
    iso=100              # ISO setting
)

# Output: Raw sensor data with Bayer pattern

Noise Models

DeepLens simulates realistic sensor noise:

  1. Shot Noise: Photon counting noise (Poisson)

  2. Dark Current Noise: Thermal electrons

  3. Read Noise: Electronic readout noise

  4. Fixed Pattern Noise: Pixel-to-pixel variation (optional)

# Enable/disable noise components
sensor = RGBSensor(
    resolution=(1920, 1080),
    enable_shot_noise=True,
    enable_dark_noise=True,
    enable_read_noise=True,
    enable_fpn=False,
    device='cuda'
)

Image Signal Processing (ISP)

Complete ISP Pipeline

from deeplens.sensor import ISP

# Create ISP pipeline
isp = ISP(
    demosaic_method='bilinear',    # or 'malvar', 'menon'
    white_balance=True,
    color_correction=True,
    gamma_correction=True,
    denoise=True,
    sharpen=False,
    device='cuda'
)

# Process raw image
rgb_image = isp(raw_image)

ISP Modules

Black Level Correction

Remove sensor pedestal:

from deeplens.sensor.isp_modules import BlackLevel

black_level = BlackLevel(level=64)  # For 12-bit sensor
corrected = black_level(raw_image)

Lens Shading Correction

Correct vignetting and non-uniform illumination:

from deeplens.sensor.isp_modules import LensShadingCorrection

lsc = LensShadingCorrection(
    resolution=(1920, 1080),
    center=[960, 540],
    falloff=0.3,
    device='cuda'
)
corrected = lsc(raw_image)

Dead Pixel Correction

Remove hot and dead pixels:

from deeplens.sensor.isp_modules import DeadPixelCorrection

dpc = DeadPixelCorrection(
    threshold=0.1,
    method='median'
)
corrected = dpc(raw_image)

White Balance

Color temperature correction:

from deeplens.sensor.isp_modules import WhiteBalance

wb = WhiteBalance(
    method='gray_world',  # or 'white_patch', 'manual'
    gains=[1.5, 1.0, 1.8]  # R, G, B gains (for manual mode)
)
balanced = wb(raw_image)

Demosaicing

Convert Bayer pattern to RGB:

from deeplens.sensor.isp_modules import Demosaic

demosaic = Demosaic(
    method='bilinear'  # or 'malvar', 'menon', 'ahd'
)
rgb = demosaic(raw_image)

Available Methods:

  • bilinear: Fast, simple interpolation

  • malvar: Edge-aware interpolation

  • menon: High-quality, edge-directed

  • ahd: Adaptive homogeneity-directed

Color Correction

Apply color correction matrix:

from deeplens.sensor.isp_modules import ColorMatrix

ccm = ColorMatrix(
    matrix=torch.tensor([
        [1.5, -0.3, -0.2],
        [-0.2, 1.3, -0.1],
        [-0.1, -0.4, 1.5]
    ])
)
corrected = ccm(rgb_image)

Gamma Correction

Apply gamma curve for display:

from deeplens.sensor.isp_modules import GammaCorrection

gamma = GammaCorrection(
    gamma=2.2,  # Standard sRGB gamma
    method='power'  # or 'srgb', 'log'
)
gamma_corrected = gamma(linear_rgb)

Denoising

Reduce noise in images:

from deeplens.sensor.isp_modules import Denoise

denoise = Denoise(
    method='bilateral',  # or 'nlm', 'bm3d'
    strength=0.5
)
denoised = denoise(rgb_image)

Color Space Conversion

Convert between color spaces:

from deeplens.sensor.isp_modules import ColorSpace

converter = ColorSpace()

# RGB to YUV
yuv = converter.rgb_to_yuv(rgb_image)

# RGB to HSV
hsv = converter.rgb_to_hsv(rgb_image)

# sRGB to linear RGB
linear = converter.srgb_to_linear(srgb_image)

Sharpening

Enhance image sharpness:

from deeplens.sensor.isp_modules import Sharpen

sharpen = Sharpen(
    strength=0.3,
    radius=1.0
)
sharpened = sharpen(rgb_image)

Anti-Aliasing

Pre-processing anti-aliasing filter:

from deeplens.sensor.isp_modules import AntiAliasing

aa = AntiAliasing(
    sigma=0.5,
    kernel_size=5
)
filtered = aa(image)

Custom ISP Pipeline

Build custom ISP pipelines:

from deeplens.sensor.isp_modules import *

class CustomISP(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.black_level = BlackLevel(level=64)
        self.lsc = LensShadingCorrection(resolution=(1920, 1080))
        self.dpc = DeadPixelCorrection()
        self.wb = WhiteBalance(method='gray_world')
        self.demosaic = Demosaic(method='malvar')
        self.ccm = ColorMatrix()
        self.gamma = GammaCorrection(gamma=2.2)
        self.denoise = Denoise(method='bilateral')

    def forward(self, raw):
        x = self.black_level(raw)
        x = self.lsc(x)
        x = self.dpc(x)
        x = self.wb(x)
        x = self.demosaic(x)
        x = self.ccm(x)
        x = self.denoise(x)
        x = self.gamma(x)
        return x

# Use custom ISP
custom_isp = CustomISP()
output = custom_isp(raw_image)

Camera System

Combining Lens and Sensor

from deeplens import Camera, GeoLens
from deeplens.sensor import RGBSensor, ISP

# Create components
lens = GeoLens(filename='./datasets/lenses/camera/ef50mm_f1.8.json')
sensor = RGBSensor(resolution=(1920, 1080), pixel_size=4.0e-3)
isp = ISP()

# Create camera
camera = Camera(
    lens=lens,
    sensor=sensor,
    isp=isp,
    device='cuda'
)

End-to-End Capture

# Load scene
import torch
from PIL import Image
import torchvision.transforms as T

scene = Image.open('./datasets/bird.png')
scene_tensor = T.ToTensor()(scene).unsqueeze(0).cuda()

# Capture image
captured_image = camera.capture(
    scene=scene_tensor,
    depth=1000.0,           # Object distance [mm]
    exposure_time=0.01,     # 10ms
    iso=100,
    auto_focus=True
)

# Save result
from torchvision.utils import save_image
save_image(captured_image, 'captured.png')

Sensor Calibration

Flat Field Correction

# Capture flat field image
flat_field = sensor.capture(uniform_illumination, exposure_time=0.01)

# Create correction map
correction_map = flat_field.mean() / (flat_field + 1e-6)

# Apply correction
corrected_image = raw_image * correction_map

Dark Frame Subtraction

# Capture dark frame (no light)
dark_frame = sensor.capture_dark(exposure_time=0.01)

# Subtract from image
corrected = raw_image - dark_frame

Sensor Response Curve

# Measure sensor response
exposures = [0.001, 0.002, 0.005, 0.01, 0.02, 0.05]
responses = []

for exp in exposures:
    raw = sensor.capture(irradiance, exposure_time=exp)
    responses.append(raw.mean())

# Plot response curve
import matplotlib.pyplot as plt
plt.plot(exposures, responses)
plt.xlabel('Exposure Time [s]')
plt.ylabel('Sensor Response [DN]')
plt.xscale('log')
plt.yscale('log')
plt.show()

Sensor Formats

Common Sensor Sizes

Format

Width (mm)

Height (mm)

Diagonal (mm)

Full Frame

36.0

24.0

43.3

APS-C (Canon)

22.2

14.8

26.7

APS-C (Nikon)

23.5

15.6

28.2

Micro 4/3

17.3

13.0

21.6

1”

13.2

8.8

15.9

1/1.7”

7.6

5.7

9.5

1/2.3”

6.2

4.6

7.7

Creating Standard Sensors

# Full frame sensor
sensor_ff = RGBSensor(
    resolution=(6000, 4000),
    sensor_size=(36.0, 24.0),
    pixel_size=6.0e-3
)

# APS-C sensor
sensor_apsc = RGBSensor(
    resolution=(6000, 4000),
    sensor_size=(23.5, 15.6),
    pixel_size=3.9e-3
)

# Smartphone sensor
sensor_phone = RGBSensor(
    resolution=(4000, 3000),
    sensor_size=(6.2, 4.6),
    pixel_size=1.55e-3
)

Performance Optimization

GPU Acceleration

# Ensure GPU usage
sensor = RGBSensor(resolution=(1920, 1080), device='cuda')

# Pre-allocate buffers
sensor.allocate_buffers()

# Batch processing
batch_size = 8
raw_batch = sensor.capture_batch(irradiance_batch, exposure_time=0.01)

Memory Management

# Process in tiles for large images
def process_tiled(image, tile_size=512):
    H, W = image.shape[-2:]
    output = torch.zeros_like(image)

    for i in range(0, H, tile_size):
        for j in range(0, W, tile_size):
            tile = image[..., i:i+tile_size, j:j+tile_size]
            output[..., i:i+tile_size, j:j+tile_size] = isp(tile)

    return output

Best Practices

Sensor Selection

  1. Pixel Size: Larger pixels → better SNR, smaller pixels → higher resolution

  2. Bit Depth: 12-14 bits sufficient for most applications

  3. Quantum Efficiency: Higher QE → better low-light performance

  4. Full Well: Larger well → higher dynamic range

ISP Pipeline Design

  1. Order Matters: Apply corrections in proper sequence

  2. Preserve Data: Use linear processing until final gamma

  3. White Balance: Apply early in pipeline for best color

  4. Denoise: Balance noise reduction vs. detail preservation

Simulation Accuracy

  1. Calibrate Sensor: Use measured parameters when available

  2. Validate Noise: Compare noise statistics with real sensor

  3. Color Accuracy: Measure and apply correct CCM

  4. Test Cases: Validate against real camera captures

Next Steps