[GIT-PULLS] [php-src] PR #23210: ext/pdo_pgsql: Add `Pdo\Pgsql::ATTR_CHUNK_SIZE` for chunked result fetching
[email protected] (KentarouTakeda)
| Newsgroups | php.git-pulls |
|---|---|
| Message-ID | <QYKKfZzyDTWDoS3C01roFVKeBtGu2OjbPvHdMvuM4wc@main.internal.php.net> |
Pull Request: https://github.com/php/php-src/pull/23210
Author: KentarouTakeda
Note: this PR currently also contains the bug fix commits from #23065. The feature is independent of them, but the existing lazy fetch defects break this feature's tests *too*. Once #23065 is merged, rebasing will leave only the last commit here.
## Summary
Adds chunked fetching through `Pdo\Pgsql::ATTR_CHUNK_SIZE`. It generalizes the lazy fetch from #15287, and the chunk size lets you choose the balance between memory and speed. ext/pgsql got this as `pg_set_chunked_rows_size` in #14571, but there was no way to use it from ext/pdo_pgsql.
## Benchmark
<details>
<summary>Benchmark code</summary>
```php
<?php
$pdo = new Pdo\Pgsql('pgsql:');
$pdo->exec(<<<SQL
drop table if exists bench;
create temp table bench as select
g as id,
md5(g::text) c1,
'user_'||g as c2,
(g%997)::int as c3,
(g*1.5)::numeric(12,2) as c4,
timestamp '2020-01-01'+(g||' seconds')::interval as c5
from
generate_series(1,500000) g;
analyze bench;
SQL);
$out = fopen('/dev/null', 'w');
$modes = [
'buffered' => [],
'ATTR_PREFETCH => 0' => [PDO::ATTR_PREFETCH => 0],
'ATTR_CHUNK_SIZE => 1' => [Pdo\Pgsql::ATTR_CHUNK_SIZE => 1],
'ATTR_CHUNK_SIZE => 10' => [Pdo\Pgsql::ATTR_CHUNK_SIZE => 10],
'ATTR_CHUNK_SIZE => 100' => [Pdo\Pgsql::ATTR_CHUNK_SIZE => 100],
'ATTR_CHUNK_SIZE => 1000' => [Pdo\Pgsql::ATTR_CHUNK_SIZE => 1000],
];
$held = $rows = $ms = [];
for ($round = -1; $round < 9; $round++) {
foreach ($modes as $name => $options) {
$t0 = hrtime(true);
$statement = $pdo->prepare('select * from bench', $options);
$statement->execute();
$held[$name] = $statement->getAttribute(Pdo\Pgsql::ATTR_RESULT_MEMORY_SIZE);
$rows[$name] = $statement->rowCount();
while ($row = $statement->fetch(PDO::FETCH_ASSOC)) {
fputcsv($out, $row, escape: '');
}
unset($statement);
// round -1 warms up
if ($round >= 0) {
$ms[$name][] = (hrtime(true) - $t0) / 1e6;
}
}
}
fputcsv(STDOUT, ['mode', 'median_ms', 'bytes_held', 'rows_per_fetch'], escape: '');
foreach ($ms as $name => $times) {
sort($times);
fputcsv(STDOUT, [$name, round($times[4]), $held[$name], $rows[$name]], escape: '');
}
```
</details>
Run on a build without `--enable-debug` (PostgreSQL 18.4 / libpq 18.4):
| mode | median_ms | bytes_held | rows_per_fetch |
| --- | ---: | ---: | ---: |
| `buffered` | 527 | 97,231,064 | 500,000 |
| `ATTR_PREFETCH => 0` | 522 | 3,288 | 1 |
| `ATTR_CHUNK_SIZE => 1` | 518 | 3,288 | 1 |
| `ATTR_CHUNK_SIZE => 10` | 484 | 3,288 | 10 |
| `ATTR_CHUNK_SIZE => 100` | 442 | 19,672 | 100 |
| `ATTR_CHUNK_SIZE => 1000` | 429 | 188,632 | 1,000 |
## Usage
Set on the connection:
```php
$pdo->setAttribute(Pdo\Pgsql::ATTR_CHUNK_SIZE, 1000);
```
Set on the statement, overriding the connection level:
```php
$pdo->prepare(
'select * from bench',
[Pdo\Pgsql::ATTR_CHUNK_SIZE => 1000]
);
```
## Design note
The part in bold is where I am not confident that the behavior is right. I would appreciate feedback.
- `ATTR_CHUNK_SIZE` overrides `ATTR_PREFETCH`.
- When `ATTR_PREFETCH` is left unset and only `ATTR_CHUNK_SIZE` is given, this is treated as a request for chunked fetching.
- **It also overrides an explicit `ATTR_PREFETCH => 0`.** The behavior changes from lazy to chunk. `ATTR_CHUNK_SIZE => 1` behaves the same as lazy, and once a chunk size is given, I think honoring it is the natural behavior.
- `ATTR_CHUNK_SIZE` and `ATTR_CURSOR => CURSOR_SCROLL` are mutually exclusive.
- A statement-level option takes precedence over the connection level, so a statement-level `CURSOR_SCROLL` disables a connection-level `ATTR_CHUNK_SIZE`.
- When both are given at the statement level, there is no basis for deciding precedence, so a `ValueError` is thrown.