[PATCH v2 2/4] media: rockchip: Add JPEG decoder driver
Sascha Hauer <[email protected]>
| Newsgroups | org.infradead.lists.linux-arm-kernel,org.infradead.lists.linux-rockchip,org.kernel.vger.linux-devicetree,org.kernel.vger.linux-kernel,org.kernel.vger.linux-media |
|---|---|
| Message-ID | <[email protected]> |
Add a driver for the JPEG hardware decoder Rockchip integrates into a number of its SoCs. Downstream it is known as the VDPU720, which is the name the register definitions carry here. Exposes one V4L2 M2M device implementing the stateful decoder interface: JPEG input -> NV12 output. The frame header is parsed on the CPU when a coded buffer is queued, which is where the resolution for V4L2_EVENT_SOURCE_CHANGE comes from; userspace cannot know the frame size without parsing the bitstream itself, so it learns it from the event and sizes its capture buffers accordingly. V4L2_DEC_CMD_STOP drains and raises V4L2_EVENT_EOS. Hardware requirements: - a DMA side buffer with Q-tables (zigzag->raster), Huffman mincode and value tables, rebuilt from the frame header on every run - a two-phase IRQ clear sequence, which some revisions require. It is done unconditionally, see rkjpegd-vdpu720-regs.h - 16-byte stream alignment with a start-byte offset - MCU-aligned PIC_H (e.g. 1080p YUV420 needs 1088, not 1080) - FILL_DOWN_E on every NV12 conversion, the output chroma is vertically subsampled so the hardware completes the bottom of the picture - DRI (restart interval) support when present - a neutral chroma plane written by the driver for a grayscale frame. The output format converter has no YUV400 path, so the hardware writes the luma plane and leaves the chroma alone, which comes out green Every address register is a single 32 bit one with no high order companion anywhere in the map, so the DMA masks are capped accordingly. That is no constraint in practice: the block sits behind the Rockchip IOMMU, whose address space is 32 bit wide by construction, and memory above 4GB is reached through it. A frame that carries no EOI marker is rejected: that is what a coded buffer too small for the frame looks like, and the hardware would otherwise hand out a half decoded picture indistinguishable from a good one. A frame larger than the negotiated capture format is rejected too, since its dimensions would be programmed against strides taken from that format. Error recovery tries the in-block soft reset first and falls back to pulsing the reset lines. The interrupt handler decodes the error status registers into a readable diagnostic. Assisted-by: Claude:claude-opus-5 Signed-off-by: Sascha Hauer <[email protected]> --- MAINTAINERS | 1 + drivers/media/platform/rockchip/Kconfig | 1 + drivers/media/platform/rockchip/Makefile | 1 + drivers/media/platform/rockchip/rkjpegd/Kconfig | 14 + drivers/media/platform/rockchip/rkjpegd/Makefile | 3 + .../rockchip/rkjpegd/rkjpegd-vdpu720-regs.h | 230 ++ drivers/media/platform/rockchip/rkjpegd/rkjpegd.c | 2328 ++++++++++++++++++++ 7 files changed, 2578 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index baea7e42663a7..9bc3dda5a4b09 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -23431,6 +23431,7 @@ L: [email protected] L: [email protected] S: Maintained F: Documentation/devicetree/bindings/media/rockchip,jpeg-decoder.yaml +F: drivers/media/platform/rockchip/rkjpegd/ ROCKCHIP RK3568 RANDOM NUMBER GENERATOR SUPPORT M: Daniel Golle <[email protected]> diff --git a/drivers/media/platform/rockchip/Kconfig b/drivers/media/platform/rockchip/Kconfig index ba401d32f01ba..53d28eb89e472 100644 --- a/drivers/media/platform/rockchip/Kconfig +++ b/drivers/media/platform/rockchip/Kconfig @@ -5,4 +5,5 @@ comment "Rockchip media platform drivers" source "drivers/media/platform/rockchip/rga/Kconfig" source "drivers/media/platform/rockchip/rkcif/Kconfig" source "drivers/media/platform/rockchip/rkisp1/Kconfig" +source "drivers/media/platform/rockchip/rkjpegd/Kconfig" source "drivers/media/platform/rockchip/rkvdec/Kconfig" diff --git a/drivers/media/platform/rockchip/Makefile b/drivers/media/platform/rockchip/Makefile index 0e0b2cbbd4bd2..c256c51ced269 100644 --- a/drivers/media/platform/rockchip/Makefile +++ b/drivers/media/platform/rockchip/Makefile @@ -2,4 +2,5 @@ obj-y += rga/ obj-y += rkcif/ obj-y += rkisp1/ +obj-y += rkjpegd/ obj-y += rkvdec/ diff --git a/drivers/media/platform/rockchip/rkjpegd/Kconfig b/drivers/media/platform/rockchip/rkjpegd/Kconfig new file mode 100644 index 0000000000000..4e2b40ef60d83 --- /dev/null +++ b/drivers/media/platform/rockchip/rkjpegd/Kconfig @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: GPL-2.0 +config VIDEO_ROCKCHIP_JPEGD + tristate "Rockchip JPEG decoder driver" + depends on ARCH_ROCKCHIP || COMPILE_TEST + depends on VIDEO_DEV + select MEDIA_CONTROLLER + select V4L2_JPEG_HELPER + select V4L2_MEM2MEM_DEV + select VIDEOBUF2_DMA_CONTIG + help + Support for the JPEG decoder Rockchip integrates into a number of + its SoCs, decoding JPEG and MJPEG frames to NV12. + To compile this driver as a module, choose M here: the module + will be called rockchip-jpegd. diff --git a/drivers/media/platform/rockchip/rkjpegd/Makefile b/drivers/media/platform/rockchip/rkjpegd/Makefile new file mode 100644 index 0000000000000..2aaf6bfcfdb74 --- /dev/null +++ b/drivers/media/platform/rockchip/rkjpegd/Makefile @@ -0,0 +1,3 @@ +obj-$(CONFIG_VIDEO_ROCKCHIP_JPEGD) += rockchip-jpegd.o + +rockchip-jpegd-y += rkjpegd.o diff --git a/drivers/media/platform/rockchip/rkjpegd/rkjpegd-vdpu720-regs.h b/drivers/media/platform/rockchip/rkjpegd/rkjpegd-vdpu720-regs.h new file mode 100644 index 0000000000000..5e2cb017cca51 --- /dev/null +++ b/drivers/media/platform/rockchip/rkjpegd/rkjpegd-vdpu720-regs.h @@ -0,0 +1,230 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Rockchip VPU720 JPEG decoder register definitions + * + * Derived from downstream Rockchip MPP HAL (hal_jpegd_rkv_reg.h). + * Copyright (C) 2020 Rockchip Electronics Co., Ltd. + * Copyright (C) 2026 WolfVision GmbH + */ +#ifndef RKJPEGD_VDPU720_REGS_H_ +#define RKJPEGD_VDPU720_REGS_H_ + +#include <linux/bitfield.h> +#include <linux/bits.h> +#include <linux/align.h> +#include <linux/types.h> + +#define VDPU720_REG_VERSION 0x000 +#define VDPU720_PROD_NUM GENMASK(31, 16) +#define VDPU720_BIT_DEPTH BIT(8) + +#define VDPU720_REG_INT 0x004 +#define VDPU720_DEC_E BIT(0) +#define VDPU720_IRQ_DIS BIT(1) +#define VDPU720_TIMEOUT_E BIT(2) +#define VDPU720_BUF_EMPTY_E BIT(3) +#define VDPU720_BUF_EMPTY_RELOAD BIT(4) +#define VDPU720_SOFT_RST_EN BIT(5) +#define VDPU720_IRQ_RAW BIT(6) +#define VDPU720_WAIT_RESET_E BIT(7) +#define VDPU720_IRQ BIT(8) +#define VDPU720_DEC_RDY BIT(9) +#define VDPU720_BUS_ERR BIT(10) +#define VDPU720_DEC_ERR BIT(11) +#define VDPU720_TIMEOUT BIT(12) +#define VDPU720_BUF_EMPTY BIT(13) +#define VDPU720_SOFT_RST_RDY BIT(14) + +/* Error status bits used to decide whether a hardware reset is needed */ +#define VDPU720_ERR_MASK (VDPU720_BUS_ERR | VDPU720_DEC_ERR | \ + VDPU720_TIMEOUT | VDPU720_BUF_EMPTY) + +/* + * VPU720 IRQ clear mask. + * + * Some revisions require a two-step IRQ acknowledgment before checking + * VDPU720_IRQ_RAW. Compute the masked value and write it back, then + * write 0 to fully clear. The downstream BSP uses: + * + * clr = (~(0x00fe7f40 & status)) & (0xff0180bf & status) + * + * Preserve "status" bits from 0xff0180bf while clearing only those + * bits NOT already set in 0x00fe7f40. + * + * It is applied unconditionally here. The BSP gates it on the revision in + * VDPU720_REG_VERSION matching 0xdb1f0006 exactly, and the RK3588 reports + * 0xdb1f0005, so it is not needed on that part. The extra write is + * harmless there and saves a revision check that would need silicon we do + * not have to validate. + */ +#define VDPU720_IRQ_CLR_KEEP 0xff0180bf +#define VDPU720_IRQ_CLR_COND 0x00fe7f40 + +#define VDPU720_REG_SYS 0x008 +#define VDPU720_FORCE_SOFTRST BIT(17) /* set when dec_e=0 before soft-reset */ +#define VDPU720_FILL_DOWN_E BIT(24) /* fill bottom padding rows */ +#define VDPU720_FILL_RIGHT_E BIT(25) +#define VDPU720_OUT_SEQ BIT(26) /* 0=raster, 1=tile */ +#define VDPU720_YUV_OUT_FMT GENMASK(29, 27) +#define VDPU720_YUV_OUT_FMT_NATIVE 0 /* no format conversion */ +#define VDPU720_YUV_OUT_FMT_NV12 3 /* output as NV12 */ + +#define VDPU720_REG_PIC_SIZE 0x00c +#define VDPU720_PIC_W_M1 GENMASK(15, 0) +#define VDPU720_PIC_H_M1 GENMASK(31, 16) + +#define VDPU720_REG_PIC_FMT 0x010 +#define VDPU720_JPEG_MODE GENMASK(2, 0) +#define VDPU720_JPEG_MODE_YUV400 0 +#define VDPU720_JPEG_MODE_YUV411 1 +#define VDPU720_JPEG_MODE_YUV420 2 +#define VDPU720_JPEG_MODE_YUV422 3 +#define VDPU720_JPEG_MODE_YUV440 4 +#define VDPU720_JPEG_MODE_YUV444 5 +#define VDPU720_PIX_DEPTH GENMASK(6, 4) +#define VDPU720_PIX_DEPTH_8 0 +#define VDPU720_PIX_DEPTH_12 1 +/* qtables_sel: number of Q-table sets (0..3) */ +#define VDPU720_QTBL_SEL GENMASK(9, 8) +/* htables_sel: number of H-table sets (0..3) */ +#define VDPU720_HTBL_SEL GENMASK(13, 12) +/* dri_e: restart interval enable */ +#define VDPU720_DRI_E BIT(15) +/* dri_mcu_num_m1: restart interval MCU count minus 1 */ +#define VDPU720_DRI_MCU_M1 GENMASK(31, 16) + +#define VDPU720_REG_HOR_STRIDE 0x014 +#define VDPU720_Y_HOR_STRIDE GENMASK(15, 0) +#define VDPU720_UV_HOR_STRIDE GENMASK(31, 16) + +#define VDPU720_REG_Y_VSTRIDE 0x018 +#define VDPU720_Y_VSTRIDE GENMASK(31, 4) + +#define VDPU720_REG_TBL_LEN 0x01c +#define VDPU720_QTBL_LEN GENMASK(4, 0) +#define VDPU720_HTBL_MINCODE_LEN GENMASK(12, 8) +#define VDPU720_HTBL_VALUE_LEN GENMASK(21, 16) +/* bit 16 of the Y horizontal stride, low 16 bits live in REG5 */ +#define VDPU720_Y_HOR_STRIDE_H BIT(24) + +#define VDPU720_REG_STRM_LEN 0x020 +#define VDPU720_STRM_START_BYTE GENMASK(3, 0) +#define VDPU720_STRM_LEN GENMASK(31, 4) + +#define VDPU720_REG_QTBL_BASE 0x024 /* Q-table side buffer, 64-byte aligned */ +#define VDPU720_REG_HTBL_MINCODE 0x028 /* H-mincode table, 64-byte aligned */ +#define VDPU720_REG_HTBL_VALUE 0x02c /* H-value table, 64-byte aligned */ +#define VDPU720_REG_STRM_BASE 0x030 /* JPEG entropy stream, 16-byte aligned */ +#define VDPU720_REG_OUT_BASE 0x034 /* NV12 output buffer, 64-byte aligned */ + +#define VDPU720_REG_STRM_ERR 0x038 +#define VDPU720_ERROR_PRC_MODE BIT(0) +#define VDPU720_STRM_FFFF_ERR_MODE GENMASK(6, 5) +#define VDPU720_STRM_OTHER_MODE GENMASK(8, 7) +/* Recommended default: accept errors, skip 0xFFFF, skip unknown markers */ +#define VDPU720_STRM_ERR_DFLT (VDPU720_ERROR_PRC_MODE | \ + FIELD_PREP_CONST(VDPU720_STRM_FFFF_ERR_MODE, 2) | \ + FIELD_PREP_CONST(VDPU720_STRM_OTHER_MODE, 2)) + +#define VDPU720_REG_CLK_GATE 0x040 +#define VDPU720_CLK_GATE_ALL 0xff + +#define VDPU720_REG_PERF_CTRL 0x078 +#define VDPU720_PERF_WORK_E BIT(0) +#define VDPU720_PERF_CLR_E BIT(1) +#define VDPU720_PERF_CNT_TYPE BIT(3) +#define VDPU720_PERF_RD_LAT_ID GENMASK(7, 4) + +#define VDPU720_REG_AXI_CFG 0x07c +#define VDPU720_ADDR_ALIGN_TYPE GENMASK(1, 0) +#define VDPU720_AR_CNT_ID_TYPE BIT(2) /* 1 = count sw_ar_count_id only */ +#define VDPU720_AW_CNT_ID_TYPE BIT(3) /* 1 = count sw_aw_count_id only */ +#define VDPU720_AR_COUNT_ID GENMASK(7, 4) +#define VDPU720_AW_COUNT_ID GENMASK(11, 8) +#define VDPU720_RD_TOTAL_BYTES_MODE BIT(12) /* 1 = count sw_ar_count_id bytes only */ + +#define VDPU720_REG_DBG_MCU_POS 0x080 +#define VDPU720_DBG_MCU_POS_X GENMASK(15, 0) /* column in MCU units */ +#define VDPU720_DBG_MCU_POS_Y GENMASK(31, 16) /* row in MCU units */ + +#define VDPU720_REG_DBG_ERROR 0x084 +#define VDPU720_DERR_DRI_SEQ BIT(0) /* DRI not at expected sequence */ +#define VDPU720_DERR_STREAM_R0 BIT(1) /* special marker 0 detected */ +#define VDPU720_DERR_STREAM_R1 BIT(2) /* special marker 1 detected */ +#define VDPU720_DERR_STREAM_FFFF BIT(3) /* 0xFFFF sequence in stream */ +#define VDPU720_DERR_OTHER_MARK BIT(4) /* unknown JPEG marker */ +#define VDPU720_DERR_MCU_CNT_L BIT(8) /* restart mark arrived too early */ +#define VDPU720_DERR_MCU_CNT_M BIT(9) /* restart mark arrived too late */ +#define VDPU720_DERR_EOI_NO_END BIT(10) /* EOI before frame complete */ +#define VDPU720_DERR_END_NO_EOI BIT(11) /* frame complete without EOI */ +#define VDPU720_DERR_OVERFLOW BIT(12) /* Huffman coefficient overflow */ +#define VDPU720_DERR_HUFF_EMPTY BIT(13) /* bitstream empty before EOI */ +#define VDPU720_DERR_FLAGS GENMASK(13, 0) /* all of the above */ +#define VDPU720_DERR_FIRST_IDX GENMASK(19, 16) /* index of first error */ + +#define VDPU720_REG_PERF_RD_MAX_LAT 0x088 /* peak read latency (clock cycles) */ +#define VDPU720_REG_PERF_RD_LAT_SAMP 0x08c /* read transactions above lat threshold */ +#define VDPU720_REG_PERF_RD_LAT_ACC 0x090 /* accumulated read latency sum */ +#define VDPU720_REG_PERF_RD_BYTES 0x094 /* total AXI read bytes this frame */ +#define VDPU720_REG_PERF_WR_BYTES 0x098 /* total AXI write bytes this frame */ + +#define VDPU720_REG_PERF_CYCLES 0x09c + +/* + * Side-buffer layout for Q-tables and Huffman tables + * + * The VPU720 JPEG decoder reads quantisation tables and Huffman + * tables from a contiguous DMA buffer with the following layout: + * + * [0, QTBL_SIZE): Q-table data (u16, raster-scan order) + * [HMINCODE_OFF, +HMIN_SZ): Huffman mincode table + * [HVALUE_OFF, +HVAL_SZ): Huffman value table + * + * The Q-tables are per component, one entry each + */ +#define VDPU720_NB_COMPONENTS 3 +#define VDPU720_QTBL_ENTRIES 64 /* 64 coefficients per table */ +#define VDPU720_QTBL_COMP_SIZE (VDPU720_QTBL_ENTRIES * sizeof(u16)) +#define VDPU720_QTBL_SIZE (VDPU720_QTBL_COMP_SIZE * VDPU720_NB_COMPONENTS) + +/* + * The Huffman tables are not per component. The hardware holds two sets and + * has no per component selector register, so the mapping is fixed: the first + * set is used for the luma component and the second one for both chroma + * components. A grayscale frame only needs the first. + */ +#define VDPU720_NB_HTBL_SETS 2 + +/* + * Per-set mincode layout: 16 DC mincodes + 8 DC accaddr pairs + + * 16 AC mincodes + 8 AC accaddr pairs = 48 u16 = 96 bytes + */ +#define VDPU720_HMINCODE_SET_SIZE (48 * sizeof(u16)) +#define VDPU720_HMINCODE_SIZE (VDPU720_HMINCODE_SET_SIZE * VDPU720_NB_HTBL_SETS) +#define VDPU720_HMINCODE_OFF VDPU720_QTBL_SIZE + +/* Per-set value layout: 16 DC values + 176 AC values = 192 bytes */ +#define VDPU720_HVALUE_SET_SIZE 192 +#define VDPU720_HVALUE_SIZE (VDPU720_HVALUE_SET_SIZE * VDPU720_NB_HTBL_SETS) +#define VDPU720_HVALUE_OFF (VDPU720_HMINCODE_OFF + \ + ALIGN(VDPU720_HMINCODE_SIZE, 64)) + +#define VDPU720_TABLE_BUF_SIZE (VDPU720_HVALUE_OFF + VDPU720_HVALUE_SIZE) + +/* + * The three table length registers count 16 byte units, minus one. Derive + * them from the sizes above so that what is programmed always matches what + * the driver writes into the side buffer. + */ +#define VDPU720_TBL_LEN_UNIT 16 +#define VDPU720_TBL_LEN(bytes) ((bytes) / VDPU720_TBL_LEN_UNIT - 1) + +/* + * Huffman value sub-layout per set (192 bytes total): + * bytes [0..15]: DC code values (up to 12 valid entries) + * bytes [16..191]: AC code values (up to 162 valid entries) + */ +#define VDPU720_DC_VALUES_MAX 16 +#define VDPU720_AC_VALUES_MAX 176 /* 12*16 - 16 */ + +#endif /* RKJPEGD_VDPU720_REGS_H_ */ diff --git a/drivers/media/platform/rockchip/rkjpegd/rkjpegd.c b/drivers/media/platform/rockchip/rkjpegd/rkjpegd.c new file mode 100644 index 0000000000000..377964eb74a64 --- /dev/null +++ b/drivers/media/platform/rockchip/rkjpegd/rkjpegd.c @@ -0,0 +1,2328 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Rockchip JPEG decoder driver + * + * The register programming is ported from the Rockchip MPP HAL + * (hal_jpegd_rkv.c / hal_jpegd_vpu7xx_com.c) and the downstream + * mpp_jpgdec.c kernel driver. + * + * Copyright (C) 2020 Rockchip Electronics Co., Ltd. + * Copyright (C) 2026 WolfVision GmbH + * Author: Lucas Sinn <[email protected]> + * Copyright (C) 2026 Pengutronix, Sascha Hauer <[email protected]> + */ + +#include <linux/align.h> +#include <linux/bitfield.h> +#include <linux/clk.h> +#include <linux/delay.h> +#include <linux/dma-mapping.h> +#include <linux/interrupt.h> +#include <linux/io.h> +#include <linux/iopoll.h> +#include <linux/log2.h> +#include <linux/module.h> +#include <linux/platform_device.h> +#include <linux/pm_runtime.h> +#include <linux/reset.h> +#include <linux/slab.h> +#include <linux/videodev2.h> +#include <linux/workqueue.h> + +#include <media/media-device.h> +#include <media/v4l2-device.h> +#include <media/v4l2-event.h> +#include <media/v4l2-fh.h> +#include <media/v4l2-ioctl.h> +#include <media/v4l2-jpeg.h> +#include <media/v4l2-mem2mem.h> +#include <media/videobuf2-core.h> +#include <media/videobuf2-dma-contig.h> + +#include "rkjpegd-vdpu720-regs.h" + +#define RKJPEGD_NAME "rockchip-jpegd" + +/* + * The reference manual gives 48x48 to 65536x65536, but a 32 bit sizeimage + * wraps well before the top of that range. Cap at four times 4K in each + * direction and hold both the coded format and the bitstream to it. + */ +#define RKJPEGD_MIN_WIDTH 48 +#define RKJPEGD_MIN_HEIGHT 48 +#define RKJPEGD_MAX_SIZE 16384 + +/* The coded queue steps in MCU-sized units, the raw one in macroblocks. */ +#define RKJPEGD_CODED_STEP 8 +#define RKJPEGD_RAW_STEP 16 + +/* Bytes per pixel used to size a coded buffer for a given resolution. */ +#define RKJPEGD_CODED_MAX_DEPTH 2 + +/* Milliseconds a frame may take before the watchdog resets the block. */ +#define RKJPEGD_TIMEOUT_MS 2000 + +static const char * const rkjpegd_clk_names[] = { + "axi", "ahb", +}; + +#define RKJPEGD_NUM_CLOCKS ARRAY_SIZE(rkjpegd_clk_names) + +/** + * struct rkjpegd_aux_buf - auxiliary DMA buffer for hardware tables + * + * @cpu: CPU pointer to the buffer. + * @dma: DMA address of the buffer. + * @size: Size of the buffer in bytes. + */ +struct rkjpegd_aux_buf { + void *cpu; + dma_addr_t dma; + size_t size; +}; + +/** + * struct rkjpegd_src_buf - a coded buffer and the header parsed out of it + * + * @base: videobuf2 mem2mem buffer, must be first. + * @header: header as returned by v4l2_jpeg_parse_header(). + * @scan: storage @header.scan points at. + * @quantization_tables: storage @header.quantization_tables points at. + * @huffman_tables: storage @header.huffman_tables points at. + * @parsed: @header describes a frame this hardware can decode. + * + * The references in @header point into the buffer payload, which stays + * mapped for as long as the buffer is queued, so the header parsed when the + * buffer arrives is still usable when the job runs. + */ +struct rkjpegd_src_buf { + struct v4l2_m2m_buffer base; + struct v4l2_jpeg_header header; + struct v4l2_jpeg_scan_header scan; + struct v4l2_jpeg_reference quantization_tables[4]; + struct v4l2_jpeg_reference huffman_tables[4]; + bool parsed; +}; + +static inline struct rkjpegd_src_buf * +vb2_to_rkjpegd_src_buf(struct vb2_buffer *vb) +{ + return container_of(to_vb2_v4l2_buffer(vb), struct rkjpegd_src_buf, + base.vb); +} + +/** + * struct rkjpegd_dev - the decoder device + * + * @v4l2_dev: V4L2 device. + * @mdev: media device. + * @vdev: video device. + * @m2m_dev: mem2mem device. + * @dev: driver model device. + * @clocks: clocks named by @rkjpegd_clk_names. + * @resets: the block's reset lines, as one array control. + * @regs: register window. + * @irq: the block's interrupt, masked while the watchdog has + * the hardware to itself. + * @vdev_lock: serialises ioctls and the videobuf2 queues. + * @watchdog_work: fires when a job does not complete in time. + */ +struct rkjpegd_dev { + struct v4l2_device v4l2_dev; + struct media_device mdev; + struct video_device vdev; + struct v4l2_m2m_dev *m2m_dev; + struct device *dev; + struct clk_bulk_data clocks[RKJPEGD_NUM_CLOCKS]; + struct reset_control *resets; + void __iomem *regs; + int irq; + struct mutex vdev_lock; /* serialises ioctls */ + struct delayed_work watchdog_work; +}; + +/** + * struct rkjpegd_ctx - one open file handle + * + * @fh: V4L2 file handle. + * @dev: device this context belongs to. + * @src_fmt: coded format on the output queue. + * @dst_fmt: raw format on the capture queue. + * @crop: visible part of a capture buffer. The decoder + * writes whole MCUs, so a frame whose width or + * height is not a multiple of one is padded, + * see vdpu720_capture_width(). + * @sequence_cap: capture buffer sequence counter. + * @sequence_out: output buffer sequence counter. + * @source_change: a resolution change was reported and the + * capture queue has not been set up again yet. + * @initial_source_change: the first parsed header must report a change + * even when it matches the negotiated format. + * @table_base: quantisation and Huffman table side buffer, + * rebuilt from the frame header on every run. + * + * @source_change and @initial_source_change, and the @dst_fmt and @crop they + * renegotiate, are written from ioctl and videobuf2 callbacks, which hold + * @rkjpegd_dev.vdev_lock, and from rkjpegd_device_run(), which does not: the + * mem2mem core runs it from its job workqueue as well as from a queue + * callback. Nothing serialises the two, but @dst_fmt and @crop are complete + * before the source change event is queued, and an application is told to + * query them only once it has dequeued that event, so one that follows the + * decoder interface cannot observe them half updated. + */ +struct rkjpegd_ctx { + struct v4l2_fh fh; + struct rkjpegd_dev *dev; + struct v4l2_pix_format_mplane src_fmt; + struct v4l2_pix_format_mplane dst_fmt; + struct v4l2_rect crop; + u32 sequence_cap; + u32 sequence_out; + bool source_change; + bool initial_source_change; + struct rkjpegd_aux_buf table_base; +}; + +static inline struct rkjpegd_ctx *file_to_rkjpegd_ctx(struct file *filp) +{ + return container_of(file_to_v4l2_fh(filp), struct rkjpegd_ctx, fh); +} + +static inline void rkjpegd_write_relaxed(struct rkjpegd_dev *jpegd, + u32 val, u32 reg) +{ + writel_relaxed(val, jpegd->regs + reg); +} + +static inline void rkjpegd_write(struct rkjpegd_dev *jpegd, u32 val, u32 reg) +{ + writel(val, jpegd->regs + reg); +} + +static inline u32 rkjpegd_read(struct rkjpegd_dev *jpegd, u32 reg) +{ + return readl(jpegd->regs + reg); +} + +static inline void rkjpegd_write_addr(struct rkjpegd_dev *jpegd, u32 reg, + dma_addr_t addr) +{ + rkjpegd_write(jpegd, lower_32_bits(addr), reg); +} + +static const struct v4l2_event rkjpegd_eos_event = { + .type = V4L2_EVENT_EOS, +}; + +static const struct v4l2_event rkjpegd_src_change_event = { + .type = V4L2_EVENT_SOURCE_CHANGE, + .u.src_change.changes = V4L2_EVENT_SRC_CH_RESOLUTION, +}; + +/* + * Colorimetry is a property of the picture, and a JPEG frame header carries + * nothing that describes it. Seed both queues with what JFIF implies and + * from then on propagate whatever userspace sets on one queue to the other, + * rather than overriding it: an application that knows the picture is BT.709 + * has no other way to say so, and one that has its format overridden takes it + * as the decoder refusing the stream. + */ +static void rkjpegd_set_default_colorimetry(struct v4l2_pix_format_mplane *pix_mp) +{ + pix_mp->colorspace = V4L2_COLORSPACE_JPEG; + pix_mp->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT; + pix_mp->quantization = V4L2_QUANTIZATION_DEFAULT; + pix_mp->xfer_func = V4L2_XFER_FUNC_DEFAULT; +} + +static void rkjpegd_propagate_colorimetry(struct v4l2_pix_format_mplane *to, + const struct v4l2_pix_format_mplane *from) +{ + to->colorspace = from->colorspace; + to->ycbcr_enc = from->ycbcr_enc; + to->quantization = from->quantization; + to->xfer_func = from->xfer_func; +} + +static void rkjpegd_fill_raw_fmt(struct v4l2_pix_format_mplane *pix_mp, + u32 width, u32 height) +{ + pix_mp->pixelformat = V4L2_PIX_FMT_NV12; + pix_mp->width = width; + pix_mp->height = height; + pix_mp->field = V4L2_FIELD_NONE; + pix_mp->num_planes = 1; + pix_mp->plane_fmt[0].bytesperline = width; + pix_mp->plane_fmt[0].sizeimage = width * height * 3 / 2; + memset(pix_mp->plane_fmt[0].reserved, 0, + sizeof(pix_mp->plane_fmt[0].reserved)); + memset(pix_mp->reserved, 0, sizeof(pix_mp->reserved)); +} + +static void rkjpegd_fill_coded_fmt(struct v4l2_pix_format_mplane *pix_mp, + u32 width, u32 height, u32 sizeimage) +{ + pix_mp->pixelformat = V4L2_PIX_FMT_JPEG; + pix_mp->width = width; + pix_mp->height = height; + pix_mp->field = V4L2_FIELD_NONE; + pix_mp->num_planes = 1; + pix_mp->plane_fmt[0].bytesperline = 0; + + /* + * A compressed frame has no size the driver could derive, so let + * userspace ask for one and only fall back to an upper bound when it + * does not. A frame that then does not fit is caught when its header + * is parsed, where the missing end of image marker gives it away. + */ + if (!sizeimage) + sizeimage = width * height * RKJPEGD_CODED_MAX_DEPTH; + pix_mp->plane_fmt[0].sizeimage = sizeimage; + + memset(pix_mp->plane_fmt[0].reserved, 0, + sizeof(pix_mp->plane_fmt[0].reserved)); + memset(pix_mp->reserved, 0, sizeof(pix_mp->reserved)); +} + +static void rkjpegd_reset_fmts(struct rkjpegd_ctx *ctx) +{ + u32 width = ALIGN(RKJPEGD_MIN_WIDTH, RKJPEGD_RAW_STEP); + u32 height = ALIGN(RKJPEGD_MIN_HEIGHT, RKJPEGD_RAW_STEP); + + rkjpegd_fill_coded_fmt(&ctx->src_fmt, width, height, 0); + rkjpegd_fill_raw_fmt(&ctx->dst_fmt, width, height); + rkjpegd_set_default_colorimetry(&ctx->src_fmt); + rkjpegd_set_default_colorimetry(&ctx->dst_fmt); + ctx->crop.left = 0; + ctx->crop.top = 0; + ctx->crop.width = width; + ctx->crop.height = height; +} + +static int rkjpegd_querycap(struct file *file, void *priv, + struct v4l2_capability *cap) +{ + strscpy(cap->driver, RKJPEGD_NAME, sizeof(cap->driver)); + strscpy(cap->card, RKJPEGD_NAME, sizeof(cap->card)); + + return 0; +} + +static int rkjpegd_enum_fmt_vid_cap(struct file *file, void *priv, + struct v4l2_fmtdesc *f) +{ + if (f->index) + return -EINVAL; + + f->pixelformat = V4L2_PIX_FMT_NV12; + + return 0; +} + +static int rkjpegd_enum_fmt_vid_out(struct file *file, void *priv, + struct v4l2_fmtdesc *f) +{ + if (f->index) + return -EINVAL; + + f->pixelformat = V4L2_PIX_FMT_JPEG; + + /* + * The frame dimensions come out of the bitstream rather than out of + * the coded format, which is what V4L2_EVENT_SOURCE_CHANGE reports. + * Userspace has no other way to find out that the event is worth + * waiting for, and one that does not wait sets the capture queue up + * from whatever the coded format happened to say. + */ + f->flags |= V4L2_FMT_FLAG_DYN_RESOLUTION; + + return 0; +} + +static int rkjpegd_enum_framesizes(struct file *file, void *priv, + struct v4l2_frmsizeenum *fsize) +{ + if (fsize->index) + return -EINVAL; + + /* + * Only the coded format has a size userspace gets to pick. The + * decoded one follows the bitstream, so there is nothing to enumerate + * for it, and saying otherwise is actively harmful: a step here + * describes the sizes that may be asked for, and userspace turns it + * into a set the visible rectangle then has to be a member of. The + * capture buffer is padded to a macroblock, the picture in it is not, + * so a 1080 line frame would fall outside a set built on a step of 16 + * and could not be negotiated at all. + */ + if (fsize->pixel_format != V4L2_PIX_FMT_JPEG) + return -ENOTTY; + + fsize->type = V4L2_FRMSIZE_TYPE_STEPWISE; + fsize->stepwise.min_width = RKJPEGD_MIN_WIDTH; + fsize->stepwise.max_width = RKJPEGD_MAX_SIZE; + fsize->stepwise.step_width = RKJPEGD_CODED_STEP; + fsize->stepwise.min_height = RKJPEGD_MIN_HEIGHT; + fsize->stepwise.max_height = RKJPEGD_MAX_SIZE; + fsize->stepwise.step_height = RKJPEGD_CODED_STEP; + + return 0; +} + +static int rkjpegd_g_fmt_vid_cap(struct file *file, void *priv, + struct v4l2_format *f) +{ + f->fmt.pix_mp = file_to_rkjpegd_ctx(file)->dst_fmt; + + return 0; +} + +static int rkjpegd_g_fmt_vid_out(struct file *file, void *priv, + struct v4l2_format *f) +{ + f->fmt.pix_mp = file_to_rkjpegd_ctx(file)->src_fmt; + + return 0; +} + +/* + * The capture resolution follows the bitstream, not userspace: it is set from + * the frame header when a source change is reported and only read back here. + */ +static int rkjpegd_try_fmt_vid_cap(struct file *file, void *priv, + struct v4l2_format *f) +{ + struct rkjpegd_ctx *ctx = file_to_rkjpegd_ctx(file); + + rkjpegd_fill_raw_fmt(&f->fmt.pix_mp, ctx->dst_fmt.width, + ctx->dst_fmt.height); + + return 0; +} + +static int rkjpegd_try_fmt_vid_out(struct file *file, void *priv, + struct v4l2_format *f) +{ + struct v4l2_pix_format_mplane *pix_mp = &f->fmt.pix_mp; + u32 sizeimage = pix_mp->num_planes == 1 ? + pix_mp->plane_fmt[0].sizeimage : 0; + + v4l_bound_align_image(&pix_mp->width, + RKJPEGD_MIN_WIDTH, RKJPEGD_MAX_SIZE, + ilog2(RKJPEGD_CODED_STEP), + &pix_mp->height, + RKJPEGD_MIN_HEIGHT, RKJPEGD_MAX_SIZE, + ilog2(RKJPEGD_CODED_STEP), 0); + + rkjpegd_fill_coded_fmt(pix_mp, pix_mp->width, pix_mp->height, sizeimage); + + return 0; +} + +static int rkjpegd_s_fmt_vid_cap(struct file *file, void *priv, + struct v4l2_format *f) +{ + struct rkjpegd_ctx *ctx = file_to_rkjpegd_ctx(file); + struct vb2_queue *vq = v4l2_m2m_get_dst_vq(ctx->fh.m2m_ctx); + + if (vb2_is_busy(vq)) + return -EBUSY; + + rkjpegd_try_fmt_vid_cap(file, priv, f); + ctx->dst_fmt = f->fmt.pix_mp; + rkjpegd_propagate_colorimetry(&ctx->src_fmt, &ctx->dst_fmt); + + return 0; +} + +static int rkjpegd_s_fmt_vid_out(struct file *file, void *priv, + struct v4l2_format *f) +{ + struct rkjpegd_ctx *ctx = file_to_rkjpegd_ctx(file); + struct vb2_queue *vq = v4l2_m2m_get_src_vq(ctx->fh.m2m_ctx); + int ret; + + if (vb2_is_busy(vq)) + return -EBUSY; + + ret = rkjpegd_try_fmt_vid_out(file, priv, f); + if (ret) + return ret; + + ctx->src_fmt = f->fmt.pix_mp; + + /* + * Picking a coded format invalidates whatever was negotiated on the + * capture queue. Seed it from the coded resolution so a userspace + * that already knows the frame size can allocate without waiting for + * the first source change. + */ + rkjpegd_fill_raw_fmt(&ctx->dst_fmt, + ALIGN(ctx->src_fmt.width, RKJPEGD_RAW_STEP), + ALIGN(ctx->src_fmt.height, RKJPEGD_RAW_STEP)); + rkjpegd_propagate_colorimetry(&ctx->dst_fmt, &ctx->src_fmt); + ctx->crop.left = 0; + ctx->crop.top = 0; + ctx->crop.width = ctx->src_fmt.width; + ctx->crop.height = ctx->src_fmt.height; + + return 0; +} + +static int rkjpegd_g_selection(struct file *file, void *priv, + struct v4l2_selection *s) +{ + struct rkjpegd_ctx *ctx = file_to_rkjpegd_ctx(file); + + if (s->type != V4L2_BUF_TYPE_VIDEO_CAPTURE && + s->type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) + return -EINVAL; + + switch (s->target) { + case V4L2_SEL_TGT_COMPOSE: + case V4L2_SEL_TGT_COMPOSE_DEFAULT: + s->r = ctx->crop; + break; + case V4L2_SEL_TGT_COMPOSE_BOUNDS: + case V4L2_SEL_TGT_COMPOSE_PADDED: + s->r.left = 0; + s->r.top = 0; + s->r.width = ctx->dst_fmt.width; + s->r.height = ctx->dst_fmt.height; + break; + default: + return -EINVAL; + } + + return 0; +} + +static int rkjpegd_subscribe_event(struct v4l2_fh *fh, + const struct v4l2_event_subscription *sub) +{ + switch (sub->type) { + case V4L2_EVENT_EOS: + return v4l2_event_subscribe(fh, sub, 0, NULL); + case V4L2_EVENT_SOURCE_CHANGE: + return v4l2_src_change_event_subscribe(fh, sub); + default: + /* The decoder takes no controls, so there is nothing else. */ + return -EINVAL; + } +} + +static void rkjpegd_set_last_buffer(struct rkjpegd_ctx *ctx) +{ + struct vb2_v4l2_buffer *next_dst_buf; + + next_dst_buf = v4l2_m2m_dst_buf_remove(ctx->fh.m2m_ctx); + if (!next_dst_buf) { + ctx->fh.m2m_ctx->is_draining = true; + ctx->fh.m2m_ctx->next_buf_last = true; + return; + } + + v4l2_m2m_last_buffer_done(ctx->fh.m2m_ctx, next_dst_buf); +} + +static int rkjpegd_decoder_cmd(struct file *file, void *priv, + struct v4l2_decoder_cmd *cmd) +{ + struct rkjpegd_ctx *ctx = file_to_rkjpegd_ctx(file); + int ret; + + ret = v4l2_m2m_ioctl_try_decoder_cmd(file, priv, cmd); + if (ret < 0) + return ret; + + if (!vb2_is_streaming(v4l2_m2m_get_src_vq(ctx->fh.m2m_ctx))) + return 0; + + ret = v4l2_m2m_ioctl_decoder_cmd(file, priv, cmd); + if (ret < 0) + return ret; + + if (cmd->cmd == V4L2_DEC_CMD_STOP && + v4l2_m2m_has_stopped(ctx->fh.m2m_ctx)) + v4l2_event_queue_fh(&ctx->fh, &rkjpegd_eos_event); + + if (cmd->cmd == V4L2_DEC_CMD_START && + v4l2_m2m_has_stopped(ctx->fh.m2m_ctx)) + vb2_clear_last_buffer_dequeued(&ctx->fh.m2m_ctx->cap_q_ctx.q); + + return 0; +} + +static const struct v4l2_ioctl_ops rkjpegd_ioctl_ops = { + .vidioc_querycap = rkjpegd_querycap, + .vidioc_enum_framesizes = rkjpegd_enum_framesizes, + + .vidioc_enum_fmt_vid_cap = rkjpegd_enum_fmt_vid_cap, + .vidioc_g_fmt_vid_cap_mplane = rkjpegd_g_fmt_vid_cap, + .vidioc_try_fmt_vid_cap_mplane = rkjpegd_try_fmt_vid_cap, + .vidioc_s_fmt_vid_cap_mplane = rkjpegd_s_fmt_vid_cap, + + .vidioc_enum_fmt_vid_out = rkjpegd_enum_fmt_vid_out, + .vidioc_g_fmt_vid_out_mplane = rkjpegd_g_fmt_vid_out, + .vidioc_try_fmt_vid_out_mplane = rkjpegd_try_fmt_vid_out, + .vidioc_s_fmt_vid_out_mplane = rkjpegd_s_fmt_vid_out, + + .vidioc_g_selection = rkjpegd_g_selection, + + .vidioc_reqbufs = v4l2_m2m_ioctl_reqbufs, + .vidioc_querybuf = v4l2_m2m_ioctl_querybuf, + .vidioc_qbuf = v4l2_m2m_ioctl_qbuf, + .vidioc_dqbuf = v4l2_m2m_ioctl_dqbuf, + .vidioc_prepare_buf = v4l2_m2m_ioctl_prepare_buf, + .vidioc_create_bufs = v4l2_m2m_ioctl_create_bufs, + .vidioc_expbuf = v4l2_m2m_ioctl_expbuf, + .vidioc_remove_bufs = v4l2_m2m_ioctl_remove_bufs, + + .vidioc_streamon = v4l2_m2m_ioctl_streamon, + .vidioc_streamoff = v4l2_m2m_ioctl_streamoff, + + .vidioc_try_decoder_cmd = v4l2_m2m_ioctl_try_decoder_cmd, + .vidioc_decoder_cmd = rkjpegd_decoder_cmd, + + .vidioc_subscribe_event = rkjpegd_subscribe_event, + .vidioc_unsubscribe_event = v4l2_event_unsubscribe, +}; + +static void rkjpegd_arm_watchdog(struct rkjpegd_dev *jpegd) +{ + schedule_delayed_work(&jpegd->watchdog_work, + msecs_to_jiffies(RKJPEGD_TIMEOUT_MS)); +} + +static void rkjpegd_job_finish_no_pm(struct rkjpegd_ctx *ctx, + enum vb2_buffer_state state) +{ + struct vb2_v4l2_buffer *src, *dst; + + src = v4l2_m2m_next_src_buf(ctx->fh.m2m_ctx); + dst = v4l2_m2m_next_dst_buf(ctx->fh.m2m_ctx); + if (WARN_ON(!src) || WARN_ON(!dst)) + return; + + src->sequence = ctx->sequence_out++; + dst->sequence = ctx->sequence_cap++; + + vb2_set_plane_payload(&dst->vb2_buf, 0, + state == VB2_BUF_STATE_DONE ? + ctx->dst_fmt.plane_fmt[0].sizeimage : 0); + + if (v4l2_m2m_is_last_draining_src_buf(ctx->fh.m2m_ctx, src)) { + dst->flags |= V4L2_BUF_FLAG_LAST; + v4l2_event_queue_fh(&ctx->fh, &rkjpegd_eos_event); + v4l2_m2m_mark_stopped(ctx->fh.m2m_ctx); + } + + v4l2_m2m_buf_done_and_job_finish(ctx->dev->m2m_dev, ctx->fh.m2m_ctx, + state); +} + +static void rkjpegd_job_finish(struct rkjpegd_ctx *ctx, + enum vb2_buffer_state state) +{ + struct rkjpegd_dev *jpegd = ctx->dev; + + clk_bulk_disable(RKJPEGD_NUM_CLOCKS, jpegd->clocks); + pm_runtime_put_autosuspend(jpegd->dev); + + rkjpegd_job_finish_no_pm(ctx, state); +} + +static void rkjpegd_irq_done(struct rkjpegd_dev *jpegd, + enum vb2_buffer_state state) +{ + struct rkjpegd_ctx *ctx = v4l2_m2m_get_curr_priv(jpegd->m2m_dev); + + if (!ctx) + return; + + if (cancel_delayed_work(&jpegd->watchdog_work)) + rkjpegd_job_finish(ctx, state); +} + +/** + * vdpu720_jpeg_mode() - map JPEG sampling factors to the hardware mode + * @frame: parsed JPEG frame header + * + * The sampling factor tuple is determined by the luma channel's factors + * relative to the maximum in the frame. Standard JFIF layouts only, + * anything else returns -EINVAL: the mode also picks the MCU height and + * therefore PIC_H, so guessing one would decode into a wrong image. + * + * Return: a VDPU720_JPEG_MODE_* value, or -EINVAL for a sampling layout + * the hardware cannot be told about. + */ +static int vdpu720_jpeg_mode(const struct v4l2_jpeg_frame_header *frame) +{ + u8 h0, v0; + + if (frame->num_components == 1) + return VDPU720_JPEG_MODE_YUV400; + + /* Component 0 always carries luma in JFIF */ + h0 = frame->component[0].horizontal_sampling_factor; + v0 = frame->component[0].vertical_sampling_factor; + + if (h0 == 1 && v0 == 1) + return VDPU720_JPEG_MODE_YUV444; + if (h0 == 2 && v0 == 1) + return VDPU720_JPEG_MODE_YUV422; + if (h0 == 2 && v0 == 2) + return VDPU720_JPEG_MODE_YUV420; + if (h0 == 4 && v0 == 1) + return VDPU720_JPEG_MODE_YUV411; + if (h0 == 1 && v0 == 2) + return VDPU720_JPEG_MODE_YUV440; + + return -EINVAL; +} + +/** + * vdpu720_mcu_width() - horizontal size of a mode's minimum coded unit + * @jpeg_mode: a VDPU720_JPEG_MODE_* value + * + * MCU_W follows the luma horizontal sampling factor, h0 * 8. YUV420 and + * YUV422 subsample the luma horizontally by 2 and YUV411 by 4, giving 16 and + * 32 pixels. The rest is 8. + * + * Return: the MCU width in pixels. + */ +static u32 vdpu720_mcu_width(int jpeg_mode) +{ + if (jpeg_mode == VDPU720_JPEG_MODE_YUV411) + return 32; + if (jpeg_mode == VDPU720_JPEG_MODE_YUV420 || + jpeg_mode == VDPU720_JPEG_MODE_YUV422) + return 16; + + return 8; +} + +/** + * vdpu720_mcu_height() - vertical size of a mode's minimum coded unit + * @jpeg_mode: a VDPU720_JPEG_MODE_* value + * + * MCU_H follows the luma vertical sampling factor, v0 * 8. YUV420 and YUV440 + * subsample the luma vertically by 2, giving 16 pixels. The rest is 8. + * + * Return: the MCU height in pixels. + */ +static u32 vdpu720_mcu_height(int jpeg_mode) +{ + if (jpeg_mode == VDPU720_JPEG_MODE_YUV420 || + jpeg_mode == VDPU720_JPEG_MODE_YUV440) + return 16; + + return 8; +} + +/** + * vdpu720_capture_width() - capture buffer width for a picture width + * @hdr: parsed JPEG header, or NULL for a frame that was not parsed + * @width: picture width in pixels + * + * A capture buffer is padded to a macroblock, but the decoder writes whole + * MCUs and a YUV411 MCU is 32 pixels wide, past the macroblock every other + * mode stops at. Pad to whichever of the two is wider so that the last MCU + * column always has a row to land in. 4:1:1 at 720 pixels is the ordinary + * case rather than a corner one: 720 is a multiple of 16 but not of 32, so a + * macroblock aligned buffer would be a full MCU column short of what the + * decoder writes. + * + * The height needs no equivalent. MCU_H tops out at the 16 pixels a + * macroblock is already aligned to. + * + * A frame the parser refused has no sampling factors to ask for, and no job + * runs for it either, so the macroblock is all it needs. + * + * Return: the buffer width in pixels. + */ +static u32 vdpu720_capture_width(const struct v4l2_jpeg_header *hdr, u32 width) +{ + int jpeg_mode; + + if (!hdr) + return ALIGN(width, RKJPEGD_RAW_STEP); + + jpeg_mode = vdpu720_jpeg_mode(&hdr->frame); + if (jpeg_mode < 0) + return ALIGN(width, RKJPEGD_RAW_STEP); + + return ALIGN(width, max_t(u32, vdpu720_mcu_width(jpeg_mode), + RKJPEGD_RAW_STEP)); +} + +/** + * vdpu720_nb_htbl_sets() - number of Huffman table sets the hardware reads + * @num_components: number of components in the scan + * + * One for a grayscale frame, two for a colour one. vdpu720_write_htbl() + * fills that many sets and vdpu720_fill_regs() sizes HTBL_SEL and the length + * registers from the same number, so the two cannot drift apart. + * + * Return: the number of table sets to write. + */ +static unsigned int vdpu720_nb_htbl_sets(unsigned int num_components) +{ + return num_components == 1 ? 1 : VDPU720_NB_HTBL_SETS; +} + +/** + * vdpu720_write_qtbl() - write all Q-tables into the DMA side buffer + * @ctx: context whose side buffer receives the tables + * @hdr: parsed JPEG header the tables are taken from + * + * Tables are stored sequentially, one per component in component order. + * Each entry is widened to u16 and reordered from JPEG zig-zag scan to + * natural raster-scan order, matching the hardware expectation. + * + * Return: 0 on success, -EINVAL if the frame refers to a quantization + * table it does not carry. + */ +static int vdpu720_write_qtbl(struct rkjpegd_ctx *ctx, + const struct v4l2_jpeg_header *hdr) +{ + struct rkjpegd_dev *jpegd = ctx->dev; + u16 *base = ctx->table_base.cpu; + unsigned int k, i; + + for (k = 0; k < hdr->frame.num_components; k++) { + u8 tq_id = hdr->frame.component[k].quantization_table_selector; + u8 qtbl[VDPU720_QTBL_ENTRIES]; + u16 *dst; + + /* + * v4l2_jpeg_parse_header() filled quantization_tables[] by + * destination selector (packed DQT segments handled); .start + * points at the 64 Qk values, already past the Pq|Tq byte. + */ + if (tq_id > 3 || !hdr->quantization_tables[tq_id].start) { + dev_err_ratelimited(jpegd->dev, + "Q-table %u not found for component %u\n", + tq_id, k); + return -EINVAL; + } + + /* + * Bulk-copy the Q-table out of the uncached source buffer once; + * the per-element zigzag reads below would otherwise each be a + * separate uncached bus transaction. + */ + memcpy(qtbl, hdr->quantization_tables[tq_id].start, sizeof(qtbl)); + dst = base + k * VDPU720_QTBL_ENTRIES; + + /* + * Reorder zigzag -> raster scan. + * v4l2_jpeg_zigzag_scan_index[z] = raster position of zigzag + * element z. JPEG Q-tables are stored in zigzag order; the + * hardware expects them in natural raster (row-major) order. + */ + for (i = 0; i < VDPU720_QTBL_ENTRIES; i++) + dst[v4l2_jpeg_zigzag_scan_index[i]] = (u16)qtbl[i]; + } + + return 0; +} + +/** + * vdpu720_compute_mincode() - build the minimum Huffman code arrays + * @bits: BITS[16], the number of codes of each length 1..16 + * @min_code: output, minimum code value per length (16 entries) + * @acc_addr: output, accumulated symbol-table address per length (16 entries) + * + * Derives the two arrays the hardware needs for one Huffman table, DC or AC, + * from that table's BITS array. Algorithm ported verbatim from + * jpegd_vpu7xx_write_htbl(). + */ +static void vdpu720_compute_mincode(const u8 *bits, u16 *min_code, u16 *acc_addr) +{ + u16 code = 0, addr = 0; + unsigned int j; + + for (j = 0; j < 16; j++) { + u16 len = bits[j]; + + if (len == 0 && j > 0) { + if (code > ((u16)(min_code[j - 1]) << 1)) + min_code[j] = code; + else + min_code[j] = (u16)(min_code[j - 1]) << 1; + } else { + min_code[j] = code; + } + + code += len; + addr += len; + acc_addr[j] = addr; + code <<= 1; + } + + /* Sentinel: set min_code[0] to the last valid code + count */ + if (bits[15]) + min_code[0] = min_code[15] + bits[15] - 1; + else + min_code[0] = min_code[15]; +} + +/** + * vdpu720_write_htbl() - fill the Huffman mincode and value sub-buffers + * @ctx: context whose side buffer receives the tables + * @hdr: parsed JPEG header the tables are taken from + * + * One set is written per vdpu720_nb_htbl_sets(), fed by the scan component + * that uses it: the first by the luma component and the second by the first + * chroma one. The hardware has no room for a third set and no per component + * selector register, so a frame whose two chroma components disagree on their + * tables cannot be described to it and is refused rather than decoded with + * the wrong table for the last component. + * + * Per-set layout in the mincode buffer: + * 16 x u16 DC min-codes + * 8 x u16 DC accumulated addresses (packed pairs) + * 16 x u16 AC min-codes + * 8 x u16 AC accumulated addresses (packed pairs) + * + * Per-set layout in the value buffer (192 bytes): + * 16 bytes DC code values + * 176 bytes AC code values + * + * Return: 0 on success, -EINVAL if the scan refers to a Huffman table it + * does not carry or needs more table sets than the hardware has. + */ +static int vdpu720_write_htbl(struct rkjpegd_ctx *ctx, + const struct v4l2_jpeg_header *hdr) +{ + struct rkjpegd_dev *jpegd = ctx->dev; + const struct v4l2_jpeg_scan_header *scan = hdr->scan; + u8 *tbl_base = ctx->table_base.cpu; + u16 *p_mincode = (u16 *)(tbl_base + VDPU720_HMINCODE_OFF); + u8 *p_value = tbl_base + VDPU720_HVALUE_OFF; + unsigned int nb_sets = vdpu720_nb_htbl_sets(scan->num_components); + unsigned int k, i; + + /* The last set is shared by every remaining component */ + for (k = nb_sets; k < scan->num_components; k++) { + if (scan->component[k].dc_entropy_coding_table_selector != + scan->component[nb_sets - 1].dc_entropy_coding_table_selector || + scan->component[k].ac_entropy_coding_table_selector != + scan->component[nb_sets - 1].ac_entropy_coding_table_selector) { + dev_err_ratelimited(jpegd->dev, + "JPEG component %u uses other Huffman tables than component %u\n", + k, nb_sets - 1); + return -EINVAL; + } + } + + for (k = 0; k < nb_sets; k++) { + u8 dc_sel = scan->component[k].dc_entropy_coding_table_selector; + u8 ac_sel = scan->component[k].ac_entropy_coding_table_selector; + u8 dc_bits[16], ac_bits[16]; + const u8 *dc_src, *ac_src, *dc_vals, *ac_vals; + unsigned int dc_huffval_len, ac_huffval_len; + u16 min_dc[16], acc_dc[16]; + u16 min_ac[16], acc_ac[16]; + + /* + * v4l2_jpeg_parse_header() filled huffman_tables[] indexed by + * (Tc << 1) | Th - Tc=0/1 (DC/AC) class, Th=0/1 (luma/chroma) + * id (packed DHT segments handled). .start points at BITS[16], + * already past the Tc|Th byte. + */ + if (dc_sel > 1 || ac_sel > 1 || + !hdr->huffman_tables[dc_sel].start || + !hdr->huffman_tables[2 | ac_sel].start) { + dev_err_ratelimited(jpegd->dev, + "H-table not found for component %u (dc=%u ac=%u)\n", + k, dc_sel, ac_sel); + return -EINVAL; + } + + /* + * Layout at .start: [BITS 16B] [HUFFVAL sum(BITS)B]. Derive the + * HUFFVAL length from BITS (a packed segment's length would + * over-count when several tables share it). + */ + dc_src = hdr->huffman_tables[dc_sel].start; + ac_src = hdr->huffman_tables[2 | ac_sel].start; + + /* + * Bulk-copy the two BITS arrays out of the uncached source; they + * are otherwise walked byte-by-byte twice (the length sum here and + * again in vdpu720_compute_mincode()). The HUFFVAL blocks stay in + * the source and are copied out in one memcpy() further down. + */ + memcpy(dc_bits, dc_src, sizeof(dc_bits)); + memcpy(ac_bits, ac_src, sizeof(ac_bits)); + dc_vals = dc_src + 16; + ac_vals = ac_src + 16; + + dc_huffval_len = 0; + for (i = 0; i < 16; i++) + dc_huffval_len += dc_bits[i]; + ac_huffval_len = 0; + for (i = 0; i < 16; i++) + ac_huffval_len += ac_bits[i]; + + /* + * The value block holds VDPU720_DC_VALUES_MAX DC and + * VDPU720_AC_VALUES_MAX AC symbols, and the accumulated + * addresses below are packed two per u16, so they have to stay + * within a byte as well. A table the hardware cannot hold + * would otherwise be truncated into it silently and decode to + * a wrong image. Both limits are covered by this check, the + * value block is the tighter of the two. + */ + if (dc_huffval_len > VDPU720_DC_VALUES_MAX || + ac_huffval_len > VDPU720_AC_VALUES_MAX) { + dev_err_ratelimited(jpegd->dev, + "JPEG Huffman table too large for component %u (dc=%u ac=%u)\n", + k, dc_huffval_len, ac_huffval_len); + return -EINVAL; + } + + vdpu720_compute_mincode(dc_bits, min_dc, acc_dc); + vdpu720_compute_mincode(ac_bits, min_ac, acc_ac); + + for (i = 0; i < 16; i++) + *p_mincode++ = min_dc[i]; + for (i = 0; i < 8; i++) + *p_mincode++ = (u16)acc_dc[2 * i] | + ((u16)acc_dc[2 * i + 1] << 8); + for (i = 0; i < 16; i++) + *p_mincode++ = min_ac[i]; + for (i = 0; i < 8; i++) + *p_mincode++ = (u16)acc_ac[2 * i] | + ((u16)acc_ac[2 * i + 1] << 8); + + /* Zero-pad the value block, then fill DC then AC values. */ + memset(p_value, 0, VDPU720_HVALUE_SET_SIZE); + memcpy(p_value, dc_vals, dc_huffval_len); + memcpy(p_value + VDPU720_DC_VALUES_MAX, ac_vals, ac_huffval_len); + p_value += VDPU720_HVALUE_SET_SIZE; + } + + return 0; +} + +/** + * vdpu720_fill_regs() - program the decoder registers for one frame + * @ctx: context the job belongs to + * @hdr: parsed JPEG header + * @tbl_dma: DMA address of the Q/Huffman table side buffer + * @strm_dma: DMA address of the entropy stream, 16-byte aligned + * @strm_start_byte: offset of the first stream byte within that word + * @strm_len_blks: stream length in 16-byte blocks, minus one + * @out_dma: DMA address of the destination NV12 buffer + * + * Return: 0 on success, negative errno for a frame the hardware cannot be + * programmed for. + */ +static int vdpu720_fill_regs(struct rkjpegd_ctx *ctx, + const struct v4l2_jpeg_header *hdr, + dma_addr_t tbl_dma, + dma_addr_t strm_dma, u32 strm_start_byte, + u32 strm_len_blks, + dma_addr_t out_dma) +{ + struct rkjpegd_dev *jpegd = ctx->dev; + u32 jpeg_width = hdr->frame.width; + u32 jpeg_height = hdr->frame.height; + u32 buf_width = ctx->dst_fmt.width; + u32 buf_height = ctx->dst_fmt.height; + u32 w_align = ALIGN(buf_width, 16); + u32 y_stride = w_align >> 4; /* units of 16 pixels */ + u32 y_vstride = y_stride * buf_height; /* sets the UV plane offset */ + u32 nb_comp = hdr->frame.num_components; + /* + * qtbl_sel = number of Q-table entries written to the side buffer, + * one per component. Using the DQT segment count is wrong when a + * camera packs all Q-tables into a single segment while there are + * three components: the hardware would only read one Q-table, leaving + * Cb/Cr with garbage. + */ + u32 qtbl_sel = nb_comp; + /* + * H-table sets: one for grayscale (luma only), two for colour (luma + + * chroma). The three table lengths below are computed from these two + * counts and the side buffer layout, so what is programmed is what + * vdpu720_write_qtbl() and vdpu720_write_htbl() actually wrote. + */ + u32 htbl_sel = vdpu720_nb_htbl_sets(nb_comp); + u32 mcu_width, mcu_height, jpeg_width_aligned, jpeg_height_aligned; + u32 qtbl_len, hmin_len, hval_len; + int jpeg_mode; + u32 reg; + + jpeg_mode = vdpu720_jpeg_mode(&hdr->frame); + if (jpeg_mode < 0) { + dev_err_ratelimited(jpegd->dev, + "unsupported JPEG sampling factors %ux%u\n", + hdr->frame.component[0].horizontal_sampling_factor, + hdr->frame.component[0].vertical_sampling_factor); + return jpeg_mode; + } + + mcu_width = vdpu720_mcu_width(jpeg_mode); + mcu_height = vdpu720_mcu_height(jpeg_mode); + jpeg_width_aligned = ALIGN(jpeg_width, mcu_width); + jpeg_height_aligned = ALIGN(jpeg_height, mcu_height); + + /* + * The picture dimensions below are taken from the bitstream while the + * strides are taken from the negotiated capture format. A frame that + * is larger than what was negotiated would make the decoder write + * beyond the capture buffer, so refuse it rather than program the + * hardware with the two sets of numbers mixed. + * + * Both are compared MCU aligned, which is the extent the decoder + * actually writes: it emits whole MCUs, and an encoder pads the right + * and bottom edges out to the same boundary. Comparing the raw width + * would let a YUV411 frame through that was negotiated against a + * macroblock rather than against its 32 pixel MCU, one MCU column + * short - vdpu720_capture_width() is what keeps that from happening + * for a capture queue this driver sized itself. + */ + if (jpeg_width_aligned > buf_width || jpeg_height_aligned > buf_height) { + dev_err_ratelimited(jpegd->dev, + "JPEG %ux%u does not fit the negotiated %ux%u\n", + jpeg_width_aligned, jpeg_height_aligned, + buf_width, buf_height); + return -EINVAL; + } + + /* + * REG2: system config - always output NV12. + * + * FILL_DOWN_E completes the bottom of the picture for the vertically + * subsampled output chroma. The reference driver sets it for every + * NV12 conversion, whatever the picture and buffer heights are. + * + * FILL_RIGHT_E does the same for the right hand edge, but only an + * 8 pixel MCU width can stop short of the buffer: a wider MCU reaches + * the edge of a buffer padded to it by vdpu720_capture_width(). + */ + reg = FIELD_PREP(VDPU720_YUV_OUT_FMT, VDPU720_YUV_OUT_FMT_NV12) | + VDPU720_FILL_DOWN_E; + if (ALIGN(jpeg_width, mcu_width) < ALIGN(jpeg_width, 16)) + reg |= VDPU720_FILL_RIGHT_E; + rkjpegd_write_relaxed(jpegd, reg, VDPU720_REG_SYS); + + /* + * REG3: picture dimensions + * + * PIC_W stays at the raw header width while PIC_H is rounded up to + * the MCU boundary. The rounding is there because the vertical MCU + * count is derived from PIC_H, see above; the reference driver + * programs the raw width here and rounding it up has not been needed. + * Either way the buffer is padded out to the MCU width, so the last + * MCU column has a row to land in. + */ + rkjpegd_write_relaxed(jpegd, + FIELD_PREP(VDPU720_PIC_W_M1, jpeg_width - 1) | + FIELD_PREP(VDPU720_PIC_H_M1, jpeg_height_aligned - 1), + VDPU720_REG_PIC_SIZE); + + /* REG4: JPEG format, Q/H table counts, restart interval */ + qtbl_len = VDPU720_TBL_LEN(qtbl_sel * VDPU720_QTBL_COMP_SIZE); + hmin_len = VDPU720_TBL_LEN(htbl_sel * VDPU720_HMINCODE_SET_SIZE); + hval_len = VDPU720_TBL_LEN(htbl_sel * VDPU720_HVALUE_SET_SIZE); + + reg = FIELD_PREP(VDPU720_JPEG_MODE, jpeg_mode) | + FIELD_PREP(VDPU720_PIX_DEPTH, VDPU720_PIX_DEPTH_8) | + FIELD_PREP(VDPU720_QTBL_SEL, qtbl_sel) | + FIELD_PREP(VDPU720_HTBL_SEL, htbl_sel); + if (hdr->restart_interval) { + reg |= VDPU720_DRI_E; + reg |= FIELD_PREP(VDPU720_DRI_MCU_M1, hdr->restart_interval - 1); + } + rkjpegd_write_relaxed(jpegd, reg, VDPU720_REG_PIC_FMT); + + /* REG5: horizontal virtual strides */ + rkjpegd_write_relaxed(jpegd, + FIELD_PREP(VDPU720_Y_HOR_STRIDE, y_stride) | + FIELD_PREP(VDPU720_UV_HOR_STRIDE, y_stride), + VDPU720_REG_HOR_STRIDE); + + /* REG6: total Y-plane size (stride-units * height) */ + rkjpegd_write_relaxed(jpegd, FIELD_PREP(VDPU720_Y_VSTRIDE, y_vstride), + VDPU720_REG_Y_VSTRIDE); + + /* REG7: table lengths + high stride bit */ + reg = FIELD_PREP(VDPU720_QTBL_LEN, qtbl_len) | + FIELD_PREP(VDPU720_HTBL_MINCODE_LEN, hmin_len) | + FIELD_PREP(VDPU720_HTBL_VALUE_LEN, hval_len) | + FIELD_PREP(VDPU720_Y_HOR_STRIDE_H, y_stride >> 16); + rkjpegd_write_relaxed(jpegd, reg, VDPU720_REG_TBL_LEN); + + /* REG8: stream length and start byte */ + rkjpegd_write_relaxed(jpegd, + FIELD_PREP(VDPU720_STRM_START_BYTE, strm_start_byte) | + FIELD_PREP(VDPU720_STRM_LEN, strm_len_blks), + VDPU720_REG_STRM_LEN); + + /* REG9-REG11: Q/H table DMA addresses (side buffer) */ + rkjpegd_write_addr(jpegd, VDPU720_REG_QTBL_BASE, tbl_dma); + rkjpegd_write_addr(jpegd, VDPU720_REG_HTBL_MINCODE, + tbl_dma + VDPU720_HMINCODE_OFF); + rkjpegd_write_addr(jpegd, VDPU720_REG_HTBL_VALUE, + tbl_dma + VDPU720_HVALUE_OFF); + + /* REG12: stream base (16-byte aligned) */ + rkjpegd_write_addr(jpegd, VDPU720_REG_STRM_BASE, strm_dma); + + /* REG13: NV12 output buffer */ + rkjpegd_write_addr(jpegd, VDPU720_REG_OUT_BASE, out_dma); + + /* REG14: stream error handling defaults */ + rkjpegd_write_relaxed(jpegd, VDPU720_STRM_ERR_DFLT, VDPU720_REG_STRM_ERR); + + /* REG16: enable all internal clock gates */ + rkjpegd_write_relaxed(jpegd, VDPU720_CLK_GATE_ALL, VDPU720_REG_CLK_GATE); + + /* REG30: AXI performance counter */ + rkjpegd_write_relaxed(jpegd, + VDPU720_PERF_WORK_E | VDPU720_PERF_CLR_E | + VDPU720_PERF_CNT_TYPE | + FIELD_PREP(VDPU720_PERF_RD_LAT_ID, 0xa), + VDPU720_REG_PERF_CTRL); + + return 0; +} + +/** + * vdpu720_fill_chroma() - write neutral chroma for a grayscale frame + * @ctx: context the job belongs to + * @dst_buf: capture buffer whose chroma plane is filled + * + * The output format converter has no YUV400 path: VDPU720_YUV_OUT_FMT_NV12 + * only covers the subsampled colour modes, and for a single component frame + * the hardware writes the luma plane and leaves the chroma plane untouched. + * + * Fill the plane here, before the hardware is started: once the decode is + * running the interrupt can complete the job and hand the buffer to + * userspace at any time. + * + * Return: 0 on success, -EINVAL if the capture buffer has no kernel mapping. + */ +static int vdpu720_fill_chroma(struct rkjpegd_ctx *ctx, + struct vb2_v4l2_buffer *dst_buf) +{ + struct rkjpegd_dev *jpegd = ctx->dev; + u32 y_size = ctx->dst_fmt.plane_fmt[0].bytesperline * ctx->dst_fmt.height; + u32 size = ctx->dst_fmt.plane_fmt[0].sizeimage; + void *dst_cpu; + + dst_cpu = vb2_plane_vaddr(&dst_buf->vb2_buf, 0); + if (!dst_cpu) { + dev_err_ratelimited(jpegd->dev, + "JPEG capture buffer has no kernel mapping\n"); + return -EINVAL; + } + + memset(dst_cpu + y_size, 0x80, size - y_size); + + return 0; +} + +/** + * rkjpegd_vdpu720_init() - allocate the table side buffer + * @ctx: context to allocate the Q/Huffman table buffer for + * + * Return: 0 on success, -ENOMEM if the allocation failed. + */ +static int rkjpegd_vdpu720_init(struct rkjpegd_ctx *ctx) +{ + struct rkjpegd_dev *jpegd = ctx->dev; + + ctx->table_base.size = VDPU720_TABLE_BUF_SIZE; + ctx->table_base.cpu = dma_alloc_noncoherent(jpegd->dev, + ctx->table_base.size, + &ctx->table_base.dma, + DMA_TO_DEVICE, GFP_KERNEL); + if (!ctx->table_base.cpu) + return -ENOMEM; + + return 0; +} + +/** + * rkjpegd_vdpu720_exit() - free the table side buffer + * @ctx: context the buffer belongs to + */ +static void rkjpegd_vdpu720_exit(struct rkjpegd_ctx *ctx) +{ + struct rkjpegd_dev *jpegd = ctx->dev; + + dma_free_noncoherent(jpegd->dev, ctx->table_base.size, + ctx->table_base.cpu, ctx->table_base.dma, + DMA_TO_DEVICE); +} + +/** + * rkjpegd_vdpu720_run() - fill the side buffer, program the registers and + * start the hardware + * @ctx: context holding the queues and the side buffer + * + * The header was parsed when the coded buffer was queued and its references + * still point into that buffer, which stays mapped until the job completes. + * + * Return: 0 with the hardware started and the watchdog armed, or a negative + * errno for a frame that cannot be decoded. + */ +static int rkjpegd_vdpu720_run(struct rkjpegd_ctx *ctx) +{ + struct rkjpegd_dev *jpegd = ctx->dev; + struct vb2_v4l2_buffer *src_buf, *dst_buf; + struct rkjpegd_src_buf *src; + const struct v4l2_jpeg_header *hdr; + dma_addr_t src_dma, dst_dma; + u32 data_offset, payload; + u32 hw_strm_off, strm_off, strm_start_byte, strm_len_blks; + int ret; + + src_buf = v4l2_m2m_next_src_buf(ctx->fh.m2m_ctx); + dst_buf = v4l2_m2m_next_dst_buf(ctx->fh.m2m_ctx); + src = vb2_to_rkjpegd_src_buf(&src_buf->vb2_buf); + + if (!src->parsed) + return -EINVAL; + + hdr = &src->header; + src_dma = vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 0); + dst_dma = vb2_dma_contig_plane_dma_addr(&dst_buf->vb2_buf, 0); + data_offset = src_buf->vb2_buf.planes[0].data_offset; + payload = vb2_get_plane_payload(&src_buf->vb2_buf, 0); + + /* + * Rebuild the Q/H-table side buffer every frame. table_base is a + * cached (dma_alloc_noncoherent) buffer and write_qtbl()/write_htbl() + * stage the source tables through cached stack buffers, so the build + * stays on cached memory; dma_sync_single_for_device() then flushes it + * to DRAM before the hardware reads it. + */ + memset(ctx->table_base.cpu, 0, ctx->table_base.size); + + ret = vdpu720_write_qtbl(ctx, hdr); + if (ret) + return ret; + + ret = vdpu720_write_htbl(ctx, hdr); + if (ret) + return ret; + + dma_sync_single_for_device(jpegd->dev, ctx->table_base.dma, + ctx->table_base.size, DMA_TO_DEVICE); + + /* + * The stream register must be 16-byte aligned. Round down to the + * nearest 16-byte boundary and record the sub-block start byte. + * + * Both are taken from the start of the plane rather than from the + * payload. data_offset is set by userspace in VIDIOC_QBUF and + * videobuf2 only rejects it when it is not smaller than bytesused, + * so it carries arbitrary low bits. Splitting an address that + * already includes it would leave STRM_BASE unaligned by those bits + * with no way to encode them, and the hardware would start reading + * from the wrong offset. + */ + strm_off = data_offset + hdr->ecs_offset; + hw_strm_off = strm_off & ~0xfU; + strm_start_byte = strm_off & 0xfU; + strm_len_blks = (ALIGN(payload - hw_strm_off, 16) - 1) >> 4; + + ret = vdpu720_fill_regs(ctx, hdr, ctx->table_base.dma, + src_dma + hw_strm_off, strm_start_byte, + strm_len_blks, dst_dma); + if (ret) + return ret; + + if (hdr->frame.num_components == 1) { + ret = vdpu720_fill_chroma(ctx, dst_buf); + if (ret) + return ret; + } + + rkjpegd_arm_watchdog(jpegd); + + /* + * Enable the buffer empty condition along with the timeout. A frame + * whose entropy data ends before the last macroblock row runs the + * decoder off the end of the stream, and with the condition masked it + * has nothing to report and waits. VDPU720_ERR_MASK already treats + * the resulting status as an error, so the handler is prepared for a + * status the hardware was never told to raise. + */ + rkjpegd_write(jpegd, + VDPU720_DEC_E | VDPU720_TIMEOUT_E | VDPU720_BUF_EMPTY_E, + VDPU720_REG_INT); + + return 0; +} + +/** + * vdpu720_soft_reset() - ask the block to reset itself + * @jpegd: device to reset + * + * Trigger the in-block soft reset and wait for it to report ready. Sleeps + * while polling, so it must not be called from atomic context. + * + * Return: 0 once the block reports the reset complete, -ETIMEDOUT if it does + * not do so within 10 ms. + */ +static int vdpu720_soft_reset(struct rkjpegd_dev *jpegd) +{ + u32 status; + int ret; + + /* + * If the decoder is idle (DEC_E=0), set FORCE_SOFTRESET_VALID + * before triggering the soft reset, per downstream BSP behaviour. + */ + status = rkjpegd_read(jpegd, VDPU720_REG_INT); + if (!(status & VDPU720_DEC_E)) + rkjpegd_write(jpegd, VDPU720_FORCE_SOFTRST, VDPU720_REG_SYS); + + rkjpegd_write(jpegd, status | VDPU720_SOFT_RST_EN, VDPU720_REG_INT); + + ret = readl_relaxed_poll_timeout(jpegd->regs + VDPU720_REG_INT, status, + status & VDPU720_SOFT_RST_RDY, + 5, 10000); + if (ret) + dev_warn(jpegd->dev, "soft reset timed out\n"); + + return ret; +} + +/** + * vdpu720_hard_reset() - pulse the block's reset lines + * @jpegd: device to reset + * + * reset_control_reset() is not usable here. The lines come from a Rockchip + * CRU, and rockchip_softrst_ops in drivers/clk/rockchip/softrst.c implements + * only .assert and .deassert, so reset_control_reset() returns -ENOTSUPP + * without touching the hardware. Drive the pulse by hand instead. + * + * Only reached from the watchdog, which runs from a workqueue, so sleeping + * between the two halves is fine. + * + * Return: 0 on success, negative errno if a reset control could not be + * asserted or deasserted. + */ +static int vdpu720_hard_reset(struct rkjpegd_dev *jpegd) +{ + int ret; + + ret = reset_control_assert(jpegd->resets); + if (ret) + return ret; + + usleep_range(10, 20); + + return reset_control_deassert(jpegd->resets); +} + +/** + * rkjpegd_vdpu720_reset() - error recovery reset called by the watchdog + * @ctx: context whose job timed out + * + * Try a soft reset first; fall back to a full hardware reset if the soft + * reset does not complete. + */ +static void rkjpegd_vdpu720_reset(struct rkjpegd_ctx *ctx) +{ + struct rkjpegd_dev *jpegd = ctx->dev; + int ret; + + ret = vdpu720_soft_reset(jpegd); + if (ret) { + dev_warn(jpegd->dev, "falling back to hard reset\n"); + + ret = vdpu720_hard_reset(jpegd); + if (ret) + dev_err(jpegd->dev, "hard reset failed: %d\n", ret); + } + + rkjpegd_write(jpegd, 0, VDPU720_REG_INT); +} + +static irqreturn_t rkjpegd_vdpu720_irq(int irq, void *dev_id) +{ + struct rkjpegd_dev *jpegd = dev_id; + enum vb2_buffer_state state; + u32 status, clr_mask; + + status = rkjpegd_read(jpegd, VDPU720_REG_INT); + + /* + * Two-phase IRQ clear. Write back a masked subset of status bits + * before checking IRQ_RAW, which some revisions require. + */ + clr_mask = (~(VDPU720_IRQ_CLR_COND & status)) & + (VDPU720_IRQ_CLR_KEEP & status); + rkjpegd_write(jpegd, clr_mask, VDPU720_REG_INT); + + if (!(status & VDPU720_IRQ_RAW)) + return IRQ_NONE; + + rkjpegd_write(jpegd, 0, VDPU720_REG_INT); + + state = (status & VDPU720_ERR_MASK) ? + VB2_BUF_STATE_ERROR : VB2_BUF_STATE_DONE; + + if (status & VDPU720_DEC_ERR) { + u32 mcu_pos = rkjpegd_read(jpegd, VDPU720_REG_DBG_MCU_POS); + u32 err_info = rkjpegd_read(jpegd, VDPU720_REG_DBG_ERROR); + + dev_warn_ratelimited(jpegd->dev, + "decode error: MCU pos=(%u,%u) flags=0x%04x [%s%s%s%s%s%s%s%s%s%s] first_idx=%u\n", + (u32)FIELD_GET(VDPU720_DBG_MCU_POS_X, mcu_pos), + (u32)FIELD_GET(VDPU720_DBG_MCU_POS_Y, mcu_pos), + (u32)FIELD_GET(VDPU720_DERR_FLAGS, err_info), + (err_info & VDPU720_DERR_DRI_SEQ) ? "dri_seq " : "", + (err_info & VDPU720_DERR_STREAM_FFFF) ? "ffff " : "", + (err_info & VDPU720_DERR_OTHER_MARK) ? "bad_mark " : "", + (err_info & VDPU720_DERR_MCU_CNT_L) ? "dri_early " : "", + (err_info & VDPU720_DERR_MCU_CNT_M) ? "dri_late " : "", + (err_info & VDPU720_DERR_EOI_NO_END) ? "eoi_early " : "", + (err_info & VDPU720_DERR_END_NO_EOI) ? "no_eoi " : "", + (err_info & VDPU720_DERR_OVERFLOW) ? "overflow " : "", + (err_info & VDPU720_DERR_HUFF_EMPTY) ? "huff_empty " : "", + (err_info & (VDPU720_DERR_STREAM_R0 | + VDPU720_DERR_STREAM_R1)) ? "stream_mark " : "", + (u32)FIELD_GET(VDPU720_DERR_FIRST_IDX, err_info)); + + rkjpegd_write(jpegd, err_info, VDPU720_REG_DBG_ERROR); + } + + rkjpegd_irq_done(jpegd, state); + + return IRQ_HANDLED; +} + +static void rkjpegd_watchdog(struct work_struct *work) +{ + struct rkjpegd_dev *jpegd = container_of(to_delayed_work(work), + struct rkjpegd_dev, + watchdog_work); + struct rkjpegd_ctx *ctx = v4l2_m2m_get_curr_priv(jpegd->m2m_dev); + + if (!ctx) + return; + + disable_irq(jpegd->irq); + + dev_err(jpegd->dev, "frame processing timed out\n"); + rkjpegd_vdpu720_reset(ctx); + + /* + * The frame is handed back as an error even if it did complete: the + * block has been reset underneath it and the interrupt telling us so + * was cleared with it. After RKJPEGD_TIMEOUT_MS that is what it is. + */ + rkjpegd_job_finish(ctx, VB2_BUF_STATE_ERROR); + + enable_irq(jpegd->irq); +} + +/** + * rkjpegd_source_change() - report a resolution change to userspace + * @ctx: context the coded buffer belongs to + * @src_buf: coded buffer whose header the new resolution is taken from + * + * Renegotiates the capture format from @src_buf and reports it, unless it + * already describes what was negotiated. Sets @rkjpegd_ctx.source_change, + * which stops rkjpegd_job_ready() from letting any further job run until the + * capture queue has been set up again. + * + * Called both when a coded buffer is queued and when one reaches the head of + * the queue, see the comments at those two call sites. + */ +static void rkjpegd_source_change(struct rkjpegd_ctx *ctx, + struct rkjpegd_src_buf *src_buf) +{ + const struct v4l2_jpeg_header *hdr = NULL; + u32 width, height, buf_width, buf_height; + + if (src_buf->parsed) { + hdr = &src_buf->header; + width = hdr->frame.width; + height = hdr->frame.height; + } else { + /* + * A frame the parser refused carries no dimensions of its own, + * and the job will hand it back with an error. Report the + * change anyway, from the coded format userspace configured. + * V4L2_FMT_FLAG_DYN_RESOLUTION tells an application to wait + * for this event before it sets the capture queue up, so one + * that never arrives leaves it waiting for a frame that cannot + * come instead of seeing the error and giving up. + */ + width = ctx->src_fmt.width; + height = ctx->src_fmt.height; + } + buf_width = vdpu720_capture_width(hdr, width); + buf_height = ALIGN(height, RKJPEGD_RAW_STEP); + + if (!ctx->initial_source_change && + ctx->dst_fmt.width == buf_width && + ctx->dst_fmt.height == buf_height && + ctx->crop.width == width && ctx->crop.height == height) + return; + + dev_dbg(ctx->dev->dev, "source change to %ux%u\n", width, height); + + rkjpegd_fill_raw_fmt(&ctx->dst_fmt, buf_width, buf_height); + ctx->crop.left = 0; + ctx->crop.top = 0; + ctx->crop.width = width; + ctx->crop.height = height; + + v4l2_event_queue_fh(&ctx->fh, &rkjpegd_src_change_event); + ctx->source_change = true; + ctx->initial_source_change = false; + + if (vb2_is_streaming(v4l2_m2m_get_dst_vq(ctx->fh.m2m_ctx))) + rkjpegd_set_last_buffer(ctx); +} + +static void rkjpegd_device_run(void *priv) +{ + struct rkjpegd_ctx *ctx = priv; + struct rkjpegd_dev *jpegd = ctx->dev; + struct vb2_v4l2_buffer *src, *dst; + int ret; + + src = v4l2_m2m_next_src_buf(ctx->fh.m2m_ctx); + dst = v4l2_m2m_next_dst_buf(ctx->fh.m2m_ctx); + if (WARN_ON(!src) || WARN_ON(!dst)) + return; + + /* + * rkjpegd_buf_queue() only looks for a resolution change while the + * coded queue is empty, so a change carried by a buffer queued behind + * others is not seen there. Look again now that this buffer has + * reached the head: everything queued before it has been decoded and + * handed back, which is the point the specification wants the change + * reported at. + * + * Doing it here rather than earlier is also what keeps the buffers + * queued behind this one correct. They are still coded frames of the + * new resolution, and they stay queued while the capture queue is set + * up again, so they are decoded into buffers that fit them. + */ + rkjpegd_source_change(ctx, vb2_to_rkjpegd_src_buf(&src->vb2_buf)); + if (ctx->source_change) { + /* + * rkjpegd_set_last_buffer() has given dst back to userspace + * with V4L2_BUF_FLAG_LAST set, so it must not be touched here. + * Finish the job without consuming src either: it is decoded + * once the capture queue has been set up for it. + */ + v4l2_m2m_job_finish(jpegd->m2m_dev, ctx->fh.m2m_ctx); + return; + } + + ret = pm_runtime_resume_and_get(jpegd->dev); + if (ret < 0) + goto err_finish; + + ret = clk_bulk_enable(RKJPEGD_NUM_CLOCKS, jpegd->clocks); + if (ret) + goto err_pm_put; + + v4l2_m2m_buf_copy_metadata(src, dst); + + ret = rkjpegd_vdpu720_run(ctx); + if (ret) + goto err_clk_disable; + + return; + +err_clk_disable: + clk_bulk_disable(RKJPEGD_NUM_CLOCKS, jpegd->clocks); +err_pm_put: + pm_runtime_put_autosuspend(jpegd->dev); +err_finish: + rkjpegd_job_finish_no_pm(ctx, VB2_BUF_STATE_ERROR); +} + +static int rkjpegd_job_ready(void *priv) +{ + struct rkjpegd_ctx *ctx = priv; + + return ctx->source_change ? 0 : 1; +} + +static const struct v4l2_m2m_ops rkjpegd_m2m_ops = { + .device_run = rkjpegd_device_run, + .job_ready = rkjpegd_job_ready, +}; + +/* + * Bitstream inspection + * + * The header is parsed when the buffer is queued rather than when the job + * runs: the resolution it carries is what a source change reports, and the + * references it hands out point into the payload, which stays mapped until + * the buffer is given back. + */ + +static bool rkjpegd_has_eoi(const void *data, u32 len) +{ + u8 tail[64]; + u32 tail_len, i; + + /* + * v4l2_jpeg_parse_header() stops at the start of scan marker and never + * looks at the entropy coded data behind it, so a frame truncated + * because it did not fit the buffer parses without an error. The + * hardware would decode as many macroblocks as it finds and hand out a + * half filled frame indistinguishable from a good one. + * + * An end of image marker cannot appear inside the entropy coded data, + * where 0xff bytes are stuffed, so finding one means the frame is + * complete. Only the last few bytes are searched, which covers a + * payload padded up to a 64 byte boundary and any short trailer. + * Looking further back is not worth it: the buffer is uncached, so a + * walk over the whole payload costs one bus transaction per byte, and + * a payload ending far behind its marker is one whose bytesused was + * never set, in which case videobuf2 substitutes the full plane length + * and a recycled buffer holds the previous frame's bytes back there + * anyway. The tail is copied out in one memcpy() for the same reason. + */ + tail_len = min_t(u32, len, sizeof(tail)); + memcpy(tail, data + len - tail_len, tail_len); + + for (i = 0; i + 1 < tail_len; i++) + if (tail[i] == 0xff && tail[i + 1] == 0xd9) + return true; + + return false; +} + +static bool rkjpegd_header_supported(struct rkjpegd_dev *jpegd, + const struct v4l2_jpeg_header *header, + u32 len) +{ + if (header->frame.width < RKJPEGD_MIN_WIDTH || + header->frame.height < RKJPEGD_MIN_HEIGHT || + header->frame.width > RKJPEGD_MAX_SIZE || + header->frame.height > RKJPEGD_MAX_SIZE) { + dev_err_ratelimited(jpegd->dev, + "unsupported JPEG picture size %ux%u, not within %ux%u..%ux%u\n", + header->frame.width, header->frame.height, + RKJPEGD_MIN_WIDTH, RKJPEGD_MIN_HEIGHT, + RKJPEGD_MAX_SIZE, RKJPEGD_MAX_SIZE); + return false; + } + + /* + * v4l2_jpeg_parse_header() accepts twelve bit samples for SOF1, but + * the register programming always asks for eight bit ones. + */ + if (header->frame.precision != 8) { + dev_err_ratelimited(jpegd->dev, + "unsupported JPEG sample precision %u\n", + header->frame.precision); + return false; + } + + if (header->frame.num_components != 1 && + header->frame.num_components != 3) { + dev_err_ratelimited(jpegd->dev, + "unsupported JPEG component count %u\n", + header->frame.num_components); + return false; + } + + if (header->scan->num_components != header->frame.num_components) { + dev_err_ratelimited(jpegd->dev, + "JPEG scan covers %u of %u components, non interleaved scans are not supported\n", + header->scan->num_components, + header->frame.num_components); + return false; + } + + if (header->ecs_offset >= len) { + dev_err_ratelimited(jpegd->dev, + "JPEG entropy coded data starts beyond the payload\n"); + return false; + } + + return true; +} + +static void rkjpegd_parse_src_buf(struct rkjpegd_ctx *ctx, + struct vb2_buffer *vb) +{ + struct rkjpegd_src_buf *src_buf = vb2_to_rkjpegd_src_buf(vb); + struct rkjpegd_dev *jpegd = ctx->dev; + u32 data_offset = vb->planes[0].data_offset; + u32 len = vb2_get_plane_payload(vb, 0); + void *data = vb2_plane_vaddr(vb, 0); + int ret; + + memset(&src_buf->header, 0, sizeof(src_buf->header)); + memset(&src_buf->scan, 0, sizeof(src_buf->scan)); + memset(src_buf->quantization_tables, 0, + sizeof(src_buf->quantization_tables)); + memset(src_buf->huffman_tables, 0, sizeof(src_buf->huffman_tables)); + src_buf->header.scan = &src_buf->scan; + src_buf->header.quantization_tables = src_buf->quantization_tables; + src_buf->header.huffman_tables = src_buf->huffman_tables; + src_buf->parsed = false; + + if (!data) { + dev_err_ratelimited(jpegd->dev, + "JPEG buffer has no kernel mapping\n"); + return; + } + + if (len <= data_offset || len - data_offset < 4) { + dev_err_ratelimited(jpegd->dev, + "JPEG payload of %u bytes is too short\n", + len); + return; + } + + data += data_offset; + len -= data_offset; + + ret = v4l2_jpeg_parse_header(data, len, &src_buf->header); + if (ret < 0) { + dev_warn_ratelimited(jpegd->dev, + "failed to parse JPEG header: %d (len=%u first_bytes=%*ph)\n", + ret, len, min_t(int, len, 8), data); + return; + } + + if (!rkjpegd_header_supported(jpegd, &src_buf->header, len)) + return; + + if (!rkjpegd_has_eoi(data, len)) { + dev_err_ratelimited(jpegd->dev, + "truncated JPEG, no end of image marker at the end of the %u byte payload (buffer too small?)\n", + len); + return; + } + + src_buf->parsed = true; +} + +static int rkjpegd_queue_setup(struct vb2_queue *vq, unsigned int *num_buffers, + unsigned int *num_planes, unsigned int sizes[], + struct device *alloc_devs[]) +{ + struct rkjpegd_ctx *ctx = vb2_get_drv_priv(vq); + struct v4l2_pix_format_mplane *pix_mp; + + pix_mp = V4L2_TYPE_IS_OUTPUT(vq->type) ? &ctx->src_fmt : &ctx->dst_fmt; + + if (*num_planes) { + if (*num_planes != 1) + return -EINVAL; + if (sizes[0] < pix_mp->plane_fmt[0].sizeimage) + return -EINVAL; + return 0; + } + + *num_planes = 1; + sizes[0] = pix_mp->plane_fmt[0].sizeimage; + + /* + * The first frame to arrive after the coded queue was set up has to + * report its resolution even when it happens to match what userspace + * guessed, otherwise an application waiting for the event before it + * allocates capture buffers never gets one. + */ + if (V4L2_TYPE_IS_OUTPUT(vq->type)) + ctx->initial_source_change = true; + + return 0; +} + +static int rkjpegd_buf_out_validate(struct vb2_buffer *vb) +{ + struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb); + + vbuf->field = V4L2_FIELD_NONE; + + return 0; +} + +static int rkjpegd_buf_prepare(struct vb2_buffer *vb) +{ + struct vb2_queue *vq = vb->vb2_queue; + struct rkjpegd_ctx *ctx = vb2_get_drv_priv(vq); + struct v4l2_pix_format_mplane *pix_mp; + + if (V4L2_TYPE_IS_OUTPUT(vq->type)) { + pix_mp = &ctx->src_fmt; + + if (vb2_plane_size(vb, 0) < pix_mp->plane_fmt[0].sizeimage) + return -EINVAL; + + return 0; + } + + pix_mp = &ctx->dst_fmt; + + /* + * A capture buffer queued into a queue that is still streaming while + * a source change is pending has the size of the previous format and + * is about to be reallocated, and no job runs until it has been. Do + * not reject it here, and do not measure it against the new format + * either, which is larger than the buffer is whenever the resolution + * grew. It carries no picture and rkjpegd_stop_streaming() hands it + * back before the queue is built again. + * + * A buffer queued while the queue is not streaming is the other case. + * It belongs to the setup for the format that was just reported, + * whether that is the first of a stream or the one answering a source + * change, and it is what the next job decodes into, so it is measured + * like any other. + */ + if (ctx->source_change && vb2_is_streaming(vq)) + return 0; + + if (vb2_plane_size(vb, 0) < pix_mp->plane_fmt[0].sizeimage) + return -EINVAL; + + return 0; +} + +static void rkjpegd_buf_queue(struct vb2_buffer *vb) +{ + struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb); + struct rkjpegd_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue); + + if (V4L2_TYPE_IS_CAPTURE(vb->vb2_queue->type)) { + if (vb2_is_streaming(vb->vb2_queue) && + v4l2_m2m_dst_buf_is_last(ctx->fh.m2m_ctx)) { + vbuf->field = V4L2_FIELD_NONE; + vbuf->sequence = ctx->sequence_cap++; + v4l2_m2m_last_buffer_done(ctx->fh.m2m_ctx, vbuf); + v4l2_event_queue_fh(&ctx->fh, &rkjpegd_eos_event); + return; + } + + v4l2_m2m_buf_queue(ctx->fh.m2m_ctx, vbuf); + return; + } + + rkjpegd_parse_src_buf(ctx, vb); + + /* + * Only look at a resolution change while nothing else is pending. A + * buffer queued behind frames that have not been decoded yet would + * otherwise renegotiate the capture queue out from under them; + * rkjpegd_device_run() picks that case up once the buffer reaches the + * head of the queue. + * + * This early report is still needed for the first buffer of a stream. + * The mem2mem core runs no job until both queues are streaming, so + * rkjpegd_device_run() cannot report the resolution an application has + * to know before it can set the capture queue up and stream it on. + */ + if (!v4l2_m2m_num_src_bufs_ready(ctx->fh.m2m_ctx)) + rkjpegd_source_change(ctx, vb2_to_rkjpegd_src_buf(vb)); + + v4l2_m2m_buf_queue(ctx->fh.m2m_ctx, vbuf); +} + +static int rkjpegd_start_streaming(struct vb2_queue *vq, unsigned int count) +{ + struct rkjpegd_ctx *ctx = vb2_get_drv_priv(vq); + + v4l2_m2m_update_start_streaming_state(ctx->fh.m2m_ctx, vq); + + if (V4L2_TYPE_IS_OUTPUT(vq->type)) { + ctx->sequence_out = 0; + } else { + ctx->sequence_cap = 0; + ctx->source_change = false; + ctx->initial_source_change = false; + } + + return 0; +} + +static void rkjpegd_stop_streaming(struct vb2_queue *vq) +{ + struct rkjpegd_ctx *ctx = vb2_get_drv_priv(vq); + struct vb2_v4l2_buffer *vbuf; + + for (;;) { + if (V4L2_TYPE_IS_OUTPUT(vq->type)) + vbuf = v4l2_m2m_src_buf_remove(ctx->fh.m2m_ctx); + else + vbuf = v4l2_m2m_dst_buf_remove(ctx->fh.m2m_ctx); + if (!vbuf) + break; + v4l2_m2m_buf_done(vbuf, VB2_BUF_STATE_ERROR); + } + + v4l2_m2m_update_stop_streaming_state(ctx->fh.m2m_ctx, vq); + + /* + * A drain that was requested before the source change was reported + * has not run yet; keep it pending across the capture queue restart. + */ + if (V4L2_TYPE_IS_CAPTURE(vq->type) && ctx->source_change && + ctx->fh.m2m_ctx->last_src_buf) + ctx->fh.m2m_ctx->is_draining = true; + + if (V4L2_TYPE_IS_OUTPUT(vq->type) && + v4l2_m2m_has_stopped(ctx->fh.m2m_ctx)) + v4l2_event_queue_fh(&ctx->fh, &rkjpegd_eos_event); +} + +static const struct vb2_ops rkjpegd_queue_ops = { + .queue_setup = rkjpegd_queue_setup, + .buf_out_validate = rkjpegd_buf_out_validate, + .buf_prepare = rkjpegd_buf_prepare, + .buf_queue = rkjpegd_buf_queue, + .start_streaming = rkjpegd_start_streaming, + .stop_streaming = rkjpegd_stop_streaming, +}; + +static int rkjpegd_queue_init(void *priv, struct vb2_queue *src_vq, + struct vb2_queue *dst_vq) +{ + struct rkjpegd_ctx *ctx = priv; + struct rkjpegd_dev *jpegd = ctx->dev; + int ret; + + src_vq->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE; + src_vq->io_modes = VB2_MMAP | VB2_DMABUF; + src_vq->drv_priv = ctx; + src_vq->ops = &rkjpegd_queue_ops; + src_vq->mem_ops = &vb2_dma_contig_memops; + src_vq->buf_struct_size = sizeof(struct rkjpegd_src_buf); + src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; + src_vq->lock = &jpegd->vdev_lock; + src_vq->dev = jpegd->v4l2_dev.dev; + + /* + * Driver does mostly sequential access, so sacrifice TLB efficiency + * for faster allocation. Both queues keep their kernel mapping: the + * quantisation and Huffman tables are read out of the coded buffer, + * and a grayscale frame leaves its chroma plane to the driver. + */ + src_vq->dma_attrs = DMA_ATTR_ALLOC_SINGLE_PAGES; + + ret = vb2_queue_init(src_vq); + if (ret) + return ret; + + dst_vq->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE; + dst_vq->io_modes = VB2_MMAP | VB2_DMABUF; + dst_vq->drv_priv = ctx; + dst_vq->ops = &rkjpegd_queue_ops; + dst_vq->mem_ops = &vb2_dma_contig_memops; + dst_vq->buf_struct_size = sizeof(struct v4l2_m2m_buffer); + dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; + dst_vq->lock = &jpegd->vdev_lock; + dst_vq->dev = jpegd->v4l2_dev.dev; + dst_vq->dma_attrs = DMA_ATTR_ALLOC_SINGLE_PAGES; + + return vb2_queue_init(dst_vq); +} + +static int rkjpegd_open(struct file *filp) +{ + struct rkjpegd_dev *jpegd = video_drvdata(filp); + struct rkjpegd_ctx *ctx; + int ret; + + ctx = kzalloc_obj(*ctx); + if (!ctx) + return -ENOMEM; + + ctx->dev = jpegd; + rkjpegd_reset_fmts(ctx); + v4l2_fh_init(&ctx->fh, video_devdata(filp)); + + ctx->fh.m2m_ctx = v4l2_m2m_ctx_init(jpegd->m2m_dev, ctx, + rkjpegd_queue_init); + if (IS_ERR(ctx->fh.m2m_ctx)) { + ret = PTR_ERR(ctx->fh.m2m_ctx); + goto err_free_ctx; + } + + ret = rkjpegd_vdpu720_init(ctx); + if (ret) + goto err_cleanup_m2m_ctx; + + v4l2_fh_add(&ctx->fh, filp); + + return 0; + +err_cleanup_m2m_ctx: + v4l2_m2m_ctx_release(ctx->fh.m2m_ctx); +err_free_ctx: + v4l2_fh_exit(&ctx->fh); + kfree(ctx); + + return ret; +} + +static int rkjpegd_release(struct file *filp) +{ + struct rkjpegd_ctx *ctx = file_to_rkjpegd_ctx(filp); + + v4l2_fh_del(&ctx->fh, filp); + v4l2_m2m_ctx_release(ctx->fh.m2m_ctx); + rkjpegd_vdpu720_exit(ctx); + v4l2_fh_exit(&ctx->fh); + kfree(ctx); + + return 0; +} + +static const struct v4l2_file_operations rkjpegd_fops = { + .owner = THIS_MODULE, + .open = rkjpegd_open, + .release = rkjpegd_release, + .poll = v4l2_m2m_fop_poll, + .unlocked_ioctl = video_ioctl2, + .mmap = v4l2_m2m_fop_mmap, +}; + +static int rkjpegd_v4l2_init(struct rkjpegd_dev *jpegd) +{ + int ret; + + ret = v4l2_device_register(jpegd->dev, &jpegd->v4l2_dev); + if (ret) { + dev_err(jpegd->dev, "failed to register V4L2 device\n"); + return ret; + } + + jpegd->m2m_dev = v4l2_m2m_init(&rkjpegd_m2m_ops); + if (IS_ERR(jpegd->m2m_dev)) { + v4l2_err(&jpegd->v4l2_dev, "failed to init mem2mem device\n"); + ret = PTR_ERR(jpegd->m2m_dev); + goto err_unregister_v4l2; + } + + jpegd->mdev.dev = jpegd->dev; + strscpy(jpegd->mdev.model, RKJPEGD_NAME, sizeof(jpegd->mdev.model)); + media_device_init(&jpegd->mdev); + jpegd->v4l2_dev.mdev = &jpegd->mdev; + + jpegd->vdev.lock = &jpegd->vdev_lock; + jpegd->vdev.v4l2_dev = &jpegd->v4l2_dev; + jpegd->vdev.fops = &rkjpegd_fops; + jpegd->vdev.release = video_device_release_empty; + jpegd->vdev.vfl_dir = VFL_DIR_M2M; + jpegd->vdev.device_caps = V4L2_CAP_STREAMING | + V4L2_CAP_VIDEO_M2M_MPLANE; + jpegd->vdev.ioctl_ops = &rkjpegd_ioctl_ops; + video_set_drvdata(&jpegd->vdev, jpegd); + strscpy(jpegd->vdev.name, RKJPEGD_NAME, sizeof(jpegd->vdev.name)); + + ret = video_register_device(&jpegd->vdev, VFL_TYPE_VIDEO, -1); + if (ret) { + v4l2_err(&jpegd->v4l2_dev, "failed to register video device\n"); + goto err_cleanup_mc; + } + + ret = v4l2_m2m_register_media_controller(jpegd->m2m_dev, &jpegd->vdev, + MEDIA_ENT_F_PROC_VIDEO_DECODER); + if (ret) { + v4l2_err(&jpegd->v4l2_dev, + "failed to init V4L2 M2M media controller\n"); + goto err_unregister_vdev; + } + + ret = media_device_register(&jpegd->mdev); + if (ret) { + v4l2_err(&jpegd->v4l2_dev, "failed to register media device\n"); + goto err_unregister_mc; + } + + return 0; + +err_unregister_mc: + v4l2_m2m_unregister_media_controller(jpegd->m2m_dev); +err_unregister_vdev: + video_unregister_device(&jpegd->vdev); +err_cleanup_mc: + media_device_cleanup(&jpegd->mdev); + v4l2_m2m_release(jpegd->m2m_dev); +err_unregister_v4l2: + v4l2_device_unregister(&jpegd->v4l2_dev); + + return ret; +} + +static void rkjpegd_v4l2_cleanup(struct rkjpegd_dev *jpegd) +{ + media_device_unregister(&jpegd->mdev); + v4l2_m2m_unregister_media_controller(jpegd->m2m_dev); + video_unregister_device(&jpegd->vdev); + media_device_cleanup(&jpegd->mdev); + v4l2_m2m_release(jpegd->m2m_dev); + v4l2_device_unregister(&jpegd->v4l2_dev); +} + +static int rkjpegd_probe(struct platform_device *pdev) +{ + struct rkjpegd_dev *jpegd; + unsigned int i; + int ret; + + jpegd = devm_kzalloc(&pdev->dev, sizeof(*jpegd), GFP_KERNEL); + if (!jpegd) + return -ENOMEM; + + jpegd->dev = &pdev->dev; + platform_set_drvdata(pdev, jpegd); + mutex_init(&jpegd->vdev_lock); + INIT_DELAYED_WORK(&jpegd->watchdog_work, rkjpegd_watchdog); + + for (i = 0; i < RKJPEGD_NUM_CLOCKS; i++) + jpegd->clocks[i].id = rkjpegd_clk_names[i]; + + ret = devm_clk_bulk_get(&pdev->dev, RKJPEGD_NUM_CLOCKS, jpegd->clocks); + if (ret) + return ret; + + jpegd->resets = devm_reset_control_array_get_exclusive(&pdev->dev); + if (IS_ERR(jpegd->resets)) + return dev_err_probe(&pdev->dev, PTR_ERR(jpegd->resets), + "failed to get resets\n"); + + jpegd->regs = devm_platform_ioremap_resource(pdev, 0); + if (IS_ERR(jpegd->regs)) + return PTR_ERR(jpegd->regs); + + ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32)); + if (ret) + return dev_err_probe(&pdev->dev, ret, "failed to set DMA mask\n"); + + jpegd->irq = platform_get_irq(pdev, 0); + if (jpegd->irq < 0) + return jpegd->irq; + + ret = devm_request_irq(&pdev->dev, jpegd->irq, rkjpegd_vdpu720_irq, 0, + dev_name(&pdev->dev), jpegd); + if (ret) + return dev_err_probe(&pdev->dev, ret, "failed to request irq\n"); + + ret = clk_bulk_prepare(RKJPEGD_NUM_CLOCKS, jpegd->clocks); + if (ret) + return dev_err_probe(&pdev->dev, ret, + "failed to prepare clocks\n"); + + pm_runtime_set_autosuspend_delay(&pdev->dev, 100); + pm_runtime_use_autosuspend(&pdev->dev); + pm_runtime_enable(&pdev->dev); + + ret = reset_control_deassert(jpegd->resets); + if (ret) { + dev_err_probe(&pdev->dev, ret, "failed to deassert resets\n"); + goto err_disable_pm; + } + + ret = rkjpegd_v4l2_init(jpegd); + if (ret) + goto err_assert_resets; + + return 0; + +err_assert_resets: + reset_control_assert(jpegd->resets); +err_disable_pm: + pm_runtime_dont_use_autosuspend(&pdev->dev); + pm_runtime_disable(&pdev->dev); + clk_bulk_unprepare(RKJPEGD_NUM_CLOCKS, jpegd->clocks); + + return ret; +} + +static void rkjpegd_remove(struct platform_device *pdev) +{ + struct rkjpegd_dev *jpegd = platform_get_drvdata(pdev); + + cancel_delayed_work_sync(&jpegd->watchdog_work); + rkjpegd_v4l2_cleanup(jpegd); + reset_control_assert(jpegd->resets); + pm_runtime_dont_use_autosuspend(&pdev->dev); + pm_runtime_disable(&pdev->dev); + clk_bulk_unprepare(RKJPEGD_NUM_CLOCKS, jpegd->clocks); +} + +/* + * The clocks are prepared once and only enabled and disabled around a job, + * which is what lets rkjpegd_job_finish() run from the interrupt handler. + * There is nothing else to do around a runtime transition: the power domain + * is handled by genpd on the device's behalf. + * + * A system transition is not the same. pm_runtime_force_suspend() calls the + * runtime suspend callback whatever the usage count says, so genpd drops the + * domain under a job that is still running: userspace is frozen by then, but + * a decode started just before the freeze is not, and it has until the + * watchdog expires to finish. Park the mem2mem queue first and wait the + * running job out. That wait is bounded by the same watchdog, which runs on + * the unfreezable system workqueue and hands the job back either way. + * + * The queue has to come back either way as well. A device whose suspend + * returned an error is never handed to the resume callback. + */ +static int rkjpegd_suspend(struct device *dev) +{ + struct rkjpegd_dev *jpegd = dev_get_drvdata(dev); + int ret; + + v4l2_m2m_suspend(jpegd->m2m_dev); + + ret = pm_runtime_force_suspend(dev); + if (ret) + v4l2_m2m_resume(jpegd->m2m_dev); + + return ret; +} + +static int rkjpegd_resume(struct device *dev) +{ + struct rkjpegd_dev *jpegd = dev_get_drvdata(dev); + int ret; + + ret = pm_runtime_force_resume(dev); + + v4l2_m2m_resume(jpegd->m2m_dev); + + return ret; +} + +static const struct dev_pm_ops rkjpegd_pm_ops = { + SYSTEM_SLEEP_PM_OPS(rkjpegd_suspend, rkjpegd_resume) +}; + +static const struct of_device_id of_rkjpegd_match[] = { + { .compatible = "rockchip,rk3568-jpegd" }, + { .compatible = "rockchip,rk3588-jpegd" }, + { /* sentinel */ } +}; +MODULE_DEVICE_TABLE(of, of_rkjpegd_match); + +static struct platform_driver rkjpegd_driver = { + .probe = rkjpegd_probe, + .remove = rkjpegd_remove, + .driver = { + .name = RKJPEGD_NAME, + .of_match_table = of_rkjpegd_match, + .pm = pm_sleep_ptr(&rkjpegd_pm_ops), + }, +}; +module_platform_driver(rkjpegd_driver); + +MODULE_DESCRIPTION("Rockchip JPEG decoder driver"); +MODULE_AUTHOR("Lucas Sinn <[email protected]>"); +MODULE_LICENSE("GPL"); -- 2.47.3