Skip to content

Commit 7566e5f

Browse files
michaelalexeevwebmichaelalexeevweb
committed
feat: version 1.1.0
- add new methods getFirst ,getLast ,getNth ,forEach - increase speed
1 parent 5c2db9d commit 7566e5f

7 files changed

Lines changed: 897 additions & 30 deletions

File tree

‎README.md‎

Lines changed: 115 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
[![MIT License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
44
[![CI](https://github.com/michaelalexeevweb/php-json-chunk/actions/workflows/ci.yml/badge.svg)](https://github.com/michaelalexeevweb/php-json-chunk/actions/workflows/ci.yml)
55

6-
Memory-efficient JSON streaming for large files in PHP. Read large JSON arrays from files in chunks, iterators, or generators without loading the full file into memory.
6+
Memory-efficient **and fast** JSON streaming for large files in PHP. Read large JSON arrays from files in chunks, iterators, or generators without loading the full file into memory — **40% faster than JSON Machine**.
77

88
Process large JSON files without running out of memory.
99

@@ -28,18 +28,48 @@ For large JSON files and large datasets, that quickly becomes inefficient or imp
2828

2929
## Comparison
3030

31-
| Library | Memory usage | Streaming |
32-
|---|---|---|
33-
| `json_decode()` | ❌ High | ❌ |
34-
| `JSON Machine` | ✅ Low | ✅ |
35-
| `PhpJsonChunk` | ✅ Low | ✅ |
31+
| Approach | Memory usage | Streaming | Speed (100k records) |
32+
|---|---|---|---|
33+
| `json_decode()` | ❌ High | ❌ | — |
34+
| `JSON Machine` | ✅ Low | ✅ | 332 ms |
35+
| `PhpJsonChunk` | ✅ **Lower** | ✅ | **207 ms** ⚡ |
3636

37-
> High-level comparison for typical large-file workflows.
37+
> **PhpJsonChunk is ~40% faster than JSON Machine and uses 50% less memory** on large JSON datasets.
38+
39+
## Performance
40+
41+
Synthetic benchmark (median of 3 runs, generated dataset with 100,000 records):
42+
43+
```
44+
Records PC time PC mem JM time JM mem Time delta Time % Speed winner
45+
10000 19.7 ms 0.15 MB 34.0 ms 0.32 MB -14.3 ms -42.0% PhpJsonChunk
46+
30000 57.5 ms 0.15 MB 100.4 ms 0.32 MB -42.9 ms -42.7% PhpJsonChunk
47+
50000 97.1 ms 0.15 MB 166.6 ms 0.32 MB -69.5 ms -41.7% PhpJsonChunk
48+
100000 191.6 ms 0.15 MB 331.9 ms 0.31 MB -140.3 ms -42.3% PhpJsonChunk
49+
```
50+
51+
How to reproduce:
52+
53+
```bash
54+
composer benchmark
55+
```
56+
57+
This runs `bin/benchmark.php` and generates benchmark JSON data on the fly.
58+
59+
You can also run with custom parameters:
60+
61+
```bash
62+
php bin/benchmark.php --runs=5 --sizes=10000,50000,100000
63+
```
64+
65+
> Benchmark results depend on hardware, PHP version, and OS. Prefer median values from multiple runs.
3866
3967
## Install
4068

69+
**Requirements:** PHP 8.1+
70+
4171
```bash
42-
composer require michaelalexeevweb/php-json-chunk:^1.0.5
72+
composer require michaelalexeevweb/php-json-chunk:^1.1.0
4373
```
4474

4575
## Quick start
@@ -195,6 +225,83 @@ foreach ($generator as $chunk) {
195225
}
196226
```
197227

228+
## Convenience Methods
229+
230+
### `getFirst()`
231+
232+
Returns the first element in the target JSON array.
233+
234+
```php
235+
<?php
236+
237+
declare(strict_types=1);
238+
239+
use PhpJsonChunk\JsonChunkReader;
240+
241+
$reader = new JsonChunkReader();
242+
243+
$first = $reader->getFirst(__DIR__ . '/data.json', keyPath: 'data');
244+
var_dump($first);
245+
```
246+
247+
### `getLast()`
248+
249+
Returns the last element in the target JSON array.
250+
251+
```php
252+
<?php
253+
254+
declare(strict_types=1);
255+
256+
use PhpJsonChunk\JsonChunkReader;
257+
258+
$reader = new JsonChunkReader();
259+
260+
$last = $reader->getLast(__DIR__ . '/data.json', keyPath: 'data');
261+
var_dump($last);
262+
```
263+
264+
### `getNth()`
265+
266+
Returns the element at a specific 0-based index.
267+
268+
```php
269+
<?php
270+
271+
declare(strict_types=1);
272+
273+
use PhpJsonChunk\JsonChunkReader;
274+
275+
$reader = new JsonChunkReader();
276+
277+
$tenth = $reader->getNth(__DIR__ . '/data.json', index: 10, keyPath: 'data');
278+
var_dump($tenth);
279+
```
280+
281+
### `forEach()`
282+
283+
Iterates through all elements and executes a callback for each one. Returns the total count processed.
284+
285+
```php
286+
<?php
287+
288+
declare(strict_types=1);
289+
290+
use PhpJsonChunk\JsonChunkReader;
291+
292+
$reader = new JsonChunkReader();
293+
294+
$total = $reader->forEach(
295+
__DIR__ . '/data.json',
296+
callback: function ($item) {
297+
echo $item['name'] . "\n";
298+
},
299+
keyPath: 'data',
300+
);
301+
302+
echo "Processed $total records\n";
303+
```
304+
198305
## Common options
199306

200307
| Option | Description |

‎bin/benchmark.php‎

Lines changed: 200 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,200 @@
1+
#!/usr/bin/env php
2+
<?php
3+
4+
declare(strict_types=1);
5+
6+
use JsonMachine\Items;
7+
use PhpJsonChunk\JsonChunkReader;
8+
9+
require_once __DIR__ . '/../vendor/autoload.php';
10+
11+
/**
12+
* @param array<int, string> $argv
13+
*
14+
* @return array{runs: int, sizes: array<int, int>}
15+
*/
16+
function parseOptions(array $argv): array
17+
{
18+
$runs = 3;
19+
$sizes = [10_000, 30_000, 50_000, 100_000];
20+
21+
foreach ($argv as $argument) {
22+
if ($argument === '--help' || $argument === '-h') {
23+
fwrite(
24+
STDOUT,
25+
"Usage: php bin/benchmark.php [--runs=3] [--sizes=10000,30000]\n",
26+
);
27+
exit(0);
28+
}
29+
30+
if (str_starts_with($argument, '--runs=')) {
31+
$parsedRuns = (int) substr($argument, strlen('--runs='));
32+
if ($parsedRuns > 0) {
33+
$runs = $parsedRuns;
34+
}
35+
continue;
36+
}
37+
38+
if (str_starts_with($argument, '--sizes=')) {
39+
$sizesRaw = substr($argument, strlen('--sizes='));
40+
$parsedSizes = [];
41+
42+
foreach (explode(',', $sizesRaw) as $sizeRaw) {
43+
$size = (int) trim($sizeRaw);
44+
if ($size > 0) {
45+
$parsedSizes[] = $size;
46+
}
47+
}
48+
49+
if ($parsedSizes !== []) {
50+
$sizes = $parsedSizes;
51+
}
52+
}
53+
}
54+
55+
return [
56+
'runs' => $runs,
57+
'sizes' => $sizes,
58+
];
59+
}
60+
61+
function buildDatasetFile(int $count): string
62+
{
63+
$path = tempnam(sys_get_temp_dir(), 'jcmp_');
64+
if ($path === false) {
65+
throw new RuntimeException('Cannot create temp file.');
66+
}
67+
68+
$fh = fopen($path, 'wb');
69+
if ($fh === false) {
70+
throw new RuntimeException('Cannot open temp file.');
71+
}
72+
73+
fwrite($fh, '{"count":' . $count . ',"data":[');
74+
for ($i = 1; $i <= $count; $i++) {
75+
if ($i > 1) {
76+
fwrite($fh, ',');
77+
}
78+
79+
fwrite($fh, sprintf(
80+
'{"id":%d,"name":"test","surname":"test","createdAt":"2023-01-01T00:00:00.000Z"}',
81+
$i,
82+
));
83+
}
84+
fwrite($fh, ']}');
85+
fclose($fh);
86+
87+
return $path;
88+
}
89+
90+
/**
91+
* @return array{elapsedMs: float, peakDeltaMb: float, total: int}
92+
*/
93+
function measurePhpJsonChunk(string $filePath): array
94+
{
95+
gc_collect_cycles();
96+
memory_reset_peak_usage();
97+
$memStart = memory_get_usage(false);
98+
$timeStart = hrtime(true);
99+
100+
$reader = new JsonChunkReader();
101+
$total = 0;
102+
103+
foreach ($reader->readGenerator($filePath, keyPath: 'data') as $item) {
104+
$total++;
105+
/** @phpstan-ignore-next-line */
106+
$_ = $item;
107+
}
108+
109+
return [
110+
'elapsedMs' => (hrtime(true) - $timeStart) / 1_000_000,
111+
'peakDeltaMb' => max(0, (memory_get_peak_usage(false) - $memStart) / 1024 / 1024),
112+
'total' => $total,
113+
];
114+
}
115+
116+
/**
117+
* @return array{elapsedMs: float, peakDeltaMb: float, total: int}
118+
*/
119+
function measureJsonMachine(string $filePath): array
120+
{
121+
gc_collect_cycles();
122+
memory_reset_peak_usage();
123+
$memStart = memory_get_usage(false);
124+
$timeStart = hrtime(true);
125+
126+
$total = 0;
127+
$items = Items::fromFile($filePath, ['pointer' => '/data']);
128+
129+
foreach ($items as $item) {
130+
$total++;
131+
/** @phpstan-ignore-next-line */
132+
$_ = $item;
133+
}
134+
135+
return [
136+
'elapsedMs' => (hrtime(true) - $timeStart) / 1_000_000,
137+
'peakDeltaMb' => max(0, (memory_get_peak_usage(false) - $memStart) / 1024 / 1024),
138+
'total' => $total,
139+
];
140+
}
141+
142+
/**
143+
* @param array<int, array{elapsedMs: float, peakDeltaMb: float, total: int}> $results
144+
*/
145+
function medianMetric(array $results, string $key): float
146+
{
147+
$values = [];
148+
149+
foreach ($results as $result) {
150+
$values[] = (float) $result[$key];
151+
}
152+
153+
sort($values);
154+
155+
return $values[(int) floor(count($values) / 2)];
156+
}
157+
158+
/**
159+
* @param array<int, int> $sizes
160+
*/
161+
function runBenchmark(int $runs, array $sizes): void
162+
{
163+
printf("Synthetic benchmark (median of %d runs)\n\n", $runs);
164+
printf("%-8s %-12s %-10s %-12s %-10s %-11s %-9s %s\n", 'Records', 'PC time', 'PC mem', 'JM time', 'JM mem', 'Time delta', 'Time %', 'Speed winner');
165+
echo str_repeat('-', 100) . "\n";
166+
167+
foreach ($sizes as $count) {
168+
$filePath = buildDatasetFile($count);
169+
170+
try {
171+
$pcResults = [];
172+
$jmResults = [];
173+
174+
for ($run = 0; $run < $runs; $run++) {
175+
$pcResults[] = measurePhpJsonChunk($filePath);
176+
$jmResults[] = measureJsonMachine($filePath);
177+
}
178+
179+
$pcTime = medianMetric($pcResults, 'elapsedMs');
180+
$pcMem = medianMetric($pcResults, 'peakDeltaMb');
181+
$jmTime = medianMetric($jmResults, 'elapsedMs');
182+
$jmMem = medianMetric($jmResults, 'peakDeltaMb');
183+
184+
$timeDeltaMs = $pcTime - $jmTime;
185+
$timeDeltaPct = $jmTime > 0 ? ($timeDeltaMs / $jmTime) * 100 : 0.0;
186+
$winner = $timeDeltaMs <= 0 ? 'PhpJsonChunk' : 'JsonMachine';
187+
188+
printf("%-8d %8.1f ms %6.2f MB %8.1f ms %6.2f MB %+9.1f ms %+7.1f%% %s\n", $count, $pcTime, $pcMem, $jmTime, $jmMem, $timeDeltaMs, $timeDeltaPct, $winner);
189+
} finally {
190+
if (is_file($filePath)) {
191+
unlink($filePath);
192+
}
193+
}
194+
}
195+
196+
echo "\nLegend: Delta = PhpJsonChunk - JSON Machine (negative = PhpJsonChunk is faster)\n";
197+
}
198+
199+
$options = parseOptions(array_slice($_SERVER['argv'], 1));
200+
runBenchmark($options['runs'], $options['sizes']);

‎composer.json‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,8 @@
2121
"require-dev": {
2222
"phpunit/phpunit": "^10.5",
2323
"phpstan/phpstan": "^2.1",
24-
"ergebnis/phpstan-rules": "^2.13"
24+
"ergebnis/phpstan-rules": "^2.13",
25+
"halaxa/json-machine": "^1.2"
2526
},
2627
"autoload": {
2728
"psr-4": {
@@ -36,7 +37,8 @@
3637
"scripts": {
3738
"test": "phpunit --exclude-group performance",
3839
"test:performance": "phpunit --group performance",
39-
"benchmark": "php bin/performance.php",
40+
"benchmark": "php bin/benchmark.php",
41+
"compare": "php bin/benchmark.php",
4042
"phpstan": "phpstan analyse --configuration=phpstan.neon.dist --memory-limit=1G"
4143
},
4244
"minimum-stability": "stable",

0 commit comments

Comments
 (0)