perf(attendees): eliminate /attendees N+1 — 83 → 36 queries (-57%)#550
perf(attendees): eliminate /attendees N+1 — 83 → 36 queries (-57%)#550smarcet wants to merge 30 commits into
Conversation
…for automatic invalidation Without this, a presentation update left stale cached data for up to 20 minutes (CacheTTL=1200s). Including the last_edited Unix timestamp in the key means any update to the presentation changes the key, so the old entry is silently ignored and ages out via TTL — no explicit Cache::forget needed anywhere.
Adds the existing ServerTimingDoctrine middleware to the /events endpoint so the response carries a Server-Timing HTTP header. Chrome DevTools renders it natively in the Network tab → Timing → Server Timing section. This is profiling-only — no behaviour change. Lets us see the breakdown of boot / db / app / total time per request on main and on any branch, so we can identify which phase is actually slow before writing any optimization.
…g header Adds these phases to the Server-Timing response header on /events: boot — LARAVEL_START to server.timing.doctrine middleware start pre — middleware start to controller method entry (auth.user etc.) controller — controller method body total db — Doctrine SQL time (already existed) serializer — $response->toArray() call only post — controller return to middleware end (response wrapping) app — total - db (existing) total — middleware start to middleware end Implementation: - Reorder route middleware so server.timing.doctrine runs BEFORE auth.user; this lets it measure auth.user as part of 'pre' time. - Controller writes 4 microtime(true) markers to Session at controller_start, serializer_start, serializer_end, controller_end. - Middleware reads markers, computes deltas, emits in header, then clears them so they don't leak to recycled workers. Profiling-only — no behavior change. Chrome DevTools Network tab → Timing → Server Timing renders each metric natively.
The previous Server-Timing 'db' metric reported ~0.4ms because:
- DBAL 2.x SQLLogger / DebugStack is deprecated and not invoked by DBAL 3.x
prepared statements (the normal query path).
- DBAL 3.x Logging\Middleware logs queries but does not include duration in
the PSR-3 context, so the previous collector saw no duration to sum.
Replace with a proper DBAL Driver Middleware that times each query at the
statement-execution level:
- QueryTimingCollector — static request-scoped accumulator (totalMs + count).
- QueryTimingMiddleware (and inner Driver/Connection/Statement wrappers) —
times every query(), exec(), and prepared-Statement::execute(). Per-query
overhead is two microtime(true) calls.
Registered globally in config/doctrine.php for all three connections. The
request lifecycle middleware (ServerTimingDoctrine) now just resets the
collector at the start of each request and reads totalMs / count at the end.
Also adds the query count to the Server-Timing header as a desc on 'db'.
…1 sources
298 queries per /events request confirms a real N+1 problem in the serializer,
but we don't know WHICH 30-per-event lazy loads are firing. Without that,
any batch-load fix is guesswork.
Enhance QueryTimingCollector to also bucket queries by normalized SQL pattern
(numeric literals + quoted strings + params collapsed to '?'). At the end of
any request that ran >= 20 queries, log the top 8 patterns (count + total ms +
sample SQL) at WARNING level so they show up in laravel.log.
Example expected output:
N+1 candidate {"count":30,"totalMs":42.1,"sample":"SELECT t0.id ... FROM PresentationSpeaker t0 WHERE t0.id = ?"}
N+1 candidate {"count":30,"totalMs":35.4,"sample":"SELECT t0.id ... FROM Member t0 WHERE t0.id = ?"}
This identifies exactly which lazy associations need to be batch-loaded.
…f 84 queries
Profiling /events on dev showed 'SELECT COUNT(MemberID)' firing 84 times per
request — the largest single N+1 source. Traced to:
PresentationSerializer::getMediaUploadsSerializerType()
-> $currentUser->isAdmin() (calls belongsToGroup)
-> $presentation->memberCanEdit($currentUser) (calls belongsToGroup multiple times)
These check the same ~8 group codes against the SAME current-user Member
instance for every presentation on the page. Each call ran a raw SELECT
COUNT(MemberID) against Group_Members.
Memoize results on the Member instance ($groupMembershipCache, unannotated
so Doctrine ignores it). The cache is per-instance and per-request — naturally
discarded when Doctrine re-hydrates the entity on the next request.
Expected impact: 84 queries -> ~8 queries on /events (one per unique group
code), saving ~85ms of DB time per request. No behaviour change.
…uery Profiling /events showed two remaining N+1 patterns directly tied to speakers: - Member SELECT by id: 56 queries / 190ms per request - Presentation_Speakers composite-key lookup: 19 queries / 26ms per request Both fire from the serializer chain (PresentationSpeaker::getFirstName() / getLastName() falling back to $this->member, and $speaker->getPresentationAssignmentOrder() looking up the assignment per (presentation, speaker)). Add a single targeted DQL right before getAllByPage returns the PagingResponse: SELECT s, m FROM PresentationSpeakerAssignment a JOIN a.speaker s LEFT JOIN s.member m WHERE a.presentation IN (:ids) That warms Doctrine's identity map with every speaker + member that the serializer is about to demand, so each subsequent getSpeaker() / getMember() call hits the identity map (zero DB). Expected impact: 75 queries collapsed into 1, ~216ms of DB time saved per /events request that contains presentations. No behaviour change — the query is read-only and side-effects only the UnitOfWork.
DQL requires that the FROM clause's root alias appears in SELECT. Previous query 'SELECT s, m FROM PresentationSpeakerAssignment a JOIN a.speaker s ...' failed with a semantical error and the preload silently no-op'd (the catch block logged the warning but the calling code continued). Change to 'SELECT a, s, m' so Doctrine hydrates the assignments, speakers, and members all into the UnitOfWork. As a bonus this also pre-populates the PresentationSpeakerAssignment entities by ID, so the per-speaker getPresentationAssignmentOrder() lookup (Presentation_Speakers composite-key pattern, 19 queries) also hits identity map.
After fixing the DQL syntax, query count only dropped 220->208 instead of the predicted ~145. Either the preload returns 0 assignments, or it loads assignments but their speaker/member fields remain proxies (so the serializer still triggers per-call lazy initialization). Add a one-shot diagnostic log right after the preload that counts: - how many assignments came back - how many unique speaker entities are reachable - how many of those speakers are initialized (not proxies) - same for members Will revert this once we know which case we're in.
…re being loaded Preload diagnostic confirmed speakers+members ARE being loaded into identity map (12 unique members, all initialized). Yet 56 Member SELECTs still fire. The 56 must therefore come from OTHER code paths (created_by, updated_by, ...). Track FROM-Member SQL queries with their bound params (up to 100 per request). Log them when the total db query count exceeds 20. The params will let us correlate Member IDs across queries and identify whether they're for: - the current_user (group/permission checks) - per-event created_by / updated_by - per-presentation creator / moderator - speakers (should be 0 if preload is working)
Doctrine wraps table identifiers in backticks (FROM `Member` t0) so the plain 'FROM Member' check missed every query. Strip backticks first.
… SELECTs per request Profiling /events captured 100 Member SELECT params; 98 of them were the SAME Member ID (the current authenticated user). Root cause: ResourceServerContext:: getCurrentUser() runs $member_repository->getByExternalId() on every single call, and serializers call it many times per request (PresentationSerializer:: getMediaUploadsSerializerType(), getSerializerType(), permission checks). Cache the resolved Member instance on the ResourceServerContext (request-scoped service). The authenticated user does not change mid-request, so the same Member is the right answer every time. The side-effects (group sync, event dispatch, field updates) only fire on the first call — which is idempotent behaviour per request anyway. Expected impact: ~98 Member queries eliminated. DB time saved depends on the network RTT to the DB but typically ~100-200ms on /events.
… 20 queries
Presentation::getSelectionStatus() ran a JOIN DQL per presentation
('SELECT sp from SummitSelectedPresentation JOIN sp.list JOIN sp.presentation
WHERE p.id = :id'). Profiling /events showed it firing 20 times per request
(once per presentation, ~twice on average).
Two changes:
1. Presentation gains $preloadedSessionSelections (transient, Doctrine-ignored).
When set by a caller via setPreloadedSessionSelections(), getSelectionStatus()
uses these rows instead of firing the DQL. Result is also memoized in
$memoizedSelectionStatus so repeated calls within a request are free.
2. DoctrineSummitEventRepository::getAllByPage adds one batch query for ALL
presentations on the page, filtered with the same constants getSelectionStatus
uses (collection=Selected, list_type=Group, list_class=Session). Results are
grouped by presentation id and pushed onto each Presentation instance.
Expected: 20 queries -> 1, plus memoization saves repeat calls within the
serializer.
…hy it has no effect 20 SummitSelectedPresentation queries still fire per request despite the preload running with no warning. Add two diagnostic logs to find where the chain breaks: 1. Repository: log how many Presentation entities received the cache via setPreloadedSessionSelections, plus the concrete class names of the first few events (in case Doctrine returns a different class than the imported models\summit\Presentation). 2. Presentation::getSelectionStatus: log a cache MISS for any presentation that falls through to the DQL fallback. If the repo logs 'fed=10' but getSelectionStatus logs 10 cache MISS, the property is being lost between setter and getter (likely a Doctrine proxy hydration issue or a different Presentation instance).
…batch-loaded assignments PresentationSpeaker::getPresentationAssignmentOrder() ran a per-call DB query via the EXTRA_LAZY $this->presentations->matching() pattern — 19 queries per /events request, one per (speaker, presentation) pair the serializer touches. Two changes: 1. PresentationSpeaker gains $preloadedAssignmentOrders (transient, unannotated so Doctrine ignores it) and setPreloadedAssignmentOrder(pid, order). getPresentationAssignmentOrder checks this cache first and only falls through to the original matching() query when the cache is unset. 2. DoctrineSummitEventRepository::getAllByPage iterates the assignments it already loaded for the speaker+member preload and pushes each (pid, order) pair into the corresponding speaker. Zero extra queries. Expected: 19 queries collapsed to 0, total query count 98 -> ~79.
… /events hydration Three remaining per-event N+1 patterns (10 queries each) collapsed in a single commit: 1. Location — was lazy-loaded per event in the serializer. Add LEFT JOIN e.location loc to the main hydration query with addSelect, so Doctrine eagerly hydrates the SummitAbstractLocation (and its JOINED subclasses like SummitVenueRoom) for every event in the same SQL round-trip. 2. Tags — was a per-event lazy load on the EXTRA_LAZY ManyToMany. Add a single 'SELECT e, t FROM SummitEvent e LEFT JOIN e.tags t WHERE e.id IN (:ids)' batch query after the main hydration, fetch-joining tags so each event's tag collection is populated. 3. PresentationMaterial — same pattern for the materials OneToMany on Presentation. Single 'SELECT p, m FROM Presentation p LEFT JOIN p.materials m WHERE p.id IN (:ids)' covers media_uploads/slides/links/ videos/etc. (they all live in the materials table). Expected: 30 queries removed, total drops from 79 to ~49 per /events request.
…gory
Two more N+1 patterns from /events:
1. Sponsors (ManyToMany Company on SummitEvent) — 10 queries per request, one
per event when the serializer iterates $event->getSponsors(). Same
fetch-join batch pattern we used for tags: SELECT e, s FROM SummitEvent e
LEFT JOIN e.sponsors s WHERE e.id IN (:ids).
2. PresentationCategory (track on Presentation) — 5 queries per request from
per-event lazy loads. Add LEFT JOIN p.category cat + addSelect('cat') to
the main hydration query so it's eagerly hydrated alongside the Presentation
subclass data.
Expected: 15 more queries removed, total drops from 61 to ~46.
… N+1 hunt The profiling work that drove the /events optimization is done. Remove the temporary diagnostic logging while keeping the actual fixes and the Server-Timing header (which remains useful for ongoing visibility). Removed: - N+1 candidate pattern logger in ServerTimingDoctrine ($dbCount>=20 trigger) - 'member queries' params dump in ServerTimingDoctrine - 'preload diagnostic' speaker+member count dump in repository - 'selection-status preload diagnostic' dump in repository - 'getSelectionStatus cache MISS' log in Presentation entity - $patterns / $memberQueries / topPatterns()/normalize() in QueryTimingCollector - bindValue() override and SQL param capture in QueryTimingMiddleware Kept: - Server-Timing HTTP header with boot/pre/controller/db/serializer/post/total - QueryTimingMiddleware DBAL Driver Middleware (accurate db ms + count) - QueryTimingCollector (now just totalMs + count) - Every actual fix from the optimization series.
Record context, profiling methodology, every fix (with symptom/root cause/ impact), what was kept, what was intentionally not fixed, and the order that produced the result. Future readers should be able to follow the same approach without rediscovering the dead ends.
…ts/{id}/attendees
Same instrumentation pattern as /events (see adr/002):
- ServerTimingDoctrine on the route (before auth.user)
- timing markers in the shared ParametrizedGetAll trait
- re-enabled temporary SQL pattern logger in QueryTimingCollector +
'N+1 candidate' log entries when db_count >= 20
Diagnostic-only — will be removed in cleanup once N+1s are identified
and fixed.
… — eliminates ~24 queries /attendees profiling showed 3 DISTINCT PresentationSpeaker patterns firing ~8 times each per request (24 total). Traced to SummitAttendeeSerializer:133 $summit->getSpeakerByMember($member), which calls getSpeakerByMemberId() which runs THREE separate queries per call (moderator check, speaker check, assistance check). Two-part fix: 1. Summit gains a request-scoped $speakerByMemberIdCache (unannotated so Doctrine ignores it) and getSpeakerByMemberId() now checks/writes the cache at every return point. By itself this is just insurance against repeated lookups within a request. 2. New Summit::preloadSpeakersByMemberIds(array $ids) runs the same 3 lookup steps but with WHERE mb.id IN (:ids), then populates the cache for every member id (with null for the ones not found). Result: 3 batch queries instead of N×3 per-attendee queries. 3. ParametrizedGetAll trait grows an optional $afterQuery callable param that fires between the data load and the toArray() call so callers can warm caches without modifying the trait body for each endpoint. 4. OAuth2SummitAttendeesApiController::getAttendeesBySummit passes a closure that collects the page's attendee member ids and invokes $summit->preloadSpeakersByMemberIds($ids) before serialization.
… reset on auth context change - DoctrineSummitEventRepository: change 'SELECT sp, p' to 'SELECT sp ... JOIN FETCH sp.presentation p' so getResult() returns SummitSelectedPresentation[] instead of mixed arrays. The prior query caused $sp->getPresentation() to fail on every request, silently falling through the try/catch and leaving the getSelectionStatus() N+1 optimization inactive. - ResourceServerContext: reset cachedCurrentUser/cachedCurrentUserResolved in setAuthorizationContext() and updateAuthContextVar() so a context change mid-request (or between requests in tests) does not return a stale member.
…rQuery hook
Collapses 4 more per-attendee N+1 patterns into 4 batch fetch-join queries:
- Notes (SummitAttendeeNote, 10 q on a 10-row page)
- Tickets (SummitAttendeeTicket, 10 q)
- Tags ManyToMany (10 q)
- Member ManyToOne fetch-join (8 q)
Each runs as 'SELECT a, X FROM SummitAttendee a LEFT JOIN a.X X WHERE a.id
IN (:ids)' which fetch-joins the inverse collection / association onto each
already-loaded SummitAttendee in the UnitOfWork. Subsequent serializer
accesses read from memory.
Expected: 63 -> ~25 queries on /summits/{id}/attendees, db time roughly
halved.
Two follow-up N+1s exposed by removing the upstream lazy loads:
1. Speaker member fetch — the preloadSpeakersByMemberIds queries joined
ps.member mb but did not addSelect('mb'), so Doctrine used the join only
for filtering and the resulting speakers had unloaded Member proxies.
Add ->addSelect('mb') to all three (moderator/speaker/assistance) batch
queries. Saves ~8 Member queries per request.
2. SummitAttendeeBadge — the per-ticket badge lookup (12 q on a 10-row page)
fired once per ticket. Extend the tickets preload from
'SELECT a, t' to 'SELECT a, t, b' with an extra LEFT JOIN t.badge b,
fetch-joining the badge alongside the ticket.
Expected: another ~20 queries removed, total drops from 49 to ~29.
- adr/002: fix heading ADR-0001 → ADR-0002 to match filename - PresentationSpeaker: add clearPreloadedAssignmentOrder(id) and clearAllPreloadedAssignmentOrders() so write paths can invalidate the preloaded assignment-order cache - PresentationSpeakerCacheTest (8 tests, no DB): covers cache hit, null order, clear-single, clear-all, and Presentation preloaded selection-status path (memoization + reset) - ResourceServerContextTest: add setAuthorizationContextResetsUserCache asserting cachedCurrentUserResolved is cleared by setAuthorizationContext()
… reset on auth context change - DoctrineSummitEventRepository: change 'SELECT sp, p' to 'SELECT sp ... JOIN FETCH sp.presentation p' so getResult() returns SummitSelectedPresentation[] instead of mixed arrays. The prior query caused $sp->getPresentation() to fail on every request, silently falling through the try/catch and leaving the getSelectionStatus() N+1 optimization inactive. - ResourceServerContext: reset cachedCurrentUser/cachedCurrentUserResolved in setAuthorizationContext() and updateAuthContextVar() so a context change mid-request (or between requests in tests) does not return a stale member.
- adr/002: fix heading ADR-0001 → ADR-0002 to match filename - PresentationSpeaker: add clearPreloadedAssignmentOrder(id) and clearAllPreloadedAssignmentOrders() so write paths can invalidate the preloaded assignment-order cache - PresentationSpeakerCacheTest (8 tests, no DB): covers cache hit, null order, clear-single, clear-all, and Presentation preloaded selection-status path (memoization + reset) - ResourceServerContextTest: add setAuthorizationContextResetsUserCache asserting cachedCurrentUserResolved is cleared by setAuthorizationContext()
The afterQuery callable hook was reverted out of the trait during cleanup, which broke the attendees controller's batch preload (it still passes a closure as the 10th arg). Restore the param + invocation between data load and toArray so OAuth2SummitAttendeesApiController::getAttendeesBySummit can warm Summit::preloadSpeakersByMemberIds + the 4 fetch-join preloads (notes / tickets+badges / tags / member) it needs.
Follows the same methodology and structure as ADR-002. Records the afterQuery trait hook (reusable for future endpoints), the Summit-level speaker memo + batch preload, and the per-attendee notes/tickets+badges/ tags/member fetch-join preloads. Includes honest reading of the result — query count down 57% but wall-clock only down 12% because DB latency now dominates over N+1 count.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
📘 OpenAPI / Swagger preview ➡️ https://OpenStackweb.github.io/summit-api/openapi/pr-550/ This page is automatically updated on each push to this PR. |
There was a problem hiding this comment.
Pull request overview
This PR targets performance on GET /api/v1/summits/{id}/attendees by collapsing serializer-driven N+1 query patterns into a small set of batch preloads, leveraging a new _getAll “afterQuery” hook to warm Doctrine associations/caches before serialization.
Changes:
- Added an optional
afterQueryhook toParametrizedGetAll::_getAll()and used it on the attendees endpoint to batch preload Notes, Tickets+Badges, Tags, and Member associations. - Added request-scoped memoization plus batch preloading for
Summit::getSpeakerByMemberId()to avoid repeated per-attendee speaker lookups. - Extended Doctrine query timing tooling to optionally bucket SQL patterns for N+1 discovery and added route-level Server-Timing instrumentation to
/attendees.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| routes/api_v1.php | Enables server.timing.doctrine middleware on the attendees listing route for profiling/timing visibility. |
| app/Models/Foundation/Summit/Summit.php | Adds per-request speaker lookup cache and a batch preloader for member→speaker resolution. |
| app/Http/Middleware/ServerTimingDoctrine.php | Adds temporary N+1 pattern logging based on collected SQL timing/pattern stats. |
| app/Http/Middleware/Doctrine/QueryTimingMiddleware.php | Passes SQL text into the timing collector for per-pattern bucketing. |
| app/Http/Middleware/Doctrine/QueryTimingCollector.php | Implements per-pattern aggregation (normalize + topPatterns) for N+1 detection. |
| app/Http/Controllers/Apis/Protected/Summit/Traits/ParametrizedGetAll.php | Adds the afterQuery hook and extra timing markers around controller/serializer phases. |
| app/Http/Controllers/Apis/Protected/Summit/OAuth2SummitAttendeesApiController.php | Uses afterQuery to batch preload associations and speaker cache before serialization. |
| adr/003-attendees-endpoint-n-plus-1-elimination.md | Documents the profiling methodology, decisions, and measured impact for /attendees. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| public function query(string $sql): DBALResult | ||
| { | ||
| $start = microtime(true); | ||
| try { | ||
| return parent::query($sql); | ||
| } finally { | ||
| QueryTimingCollector::record($start); | ||
| QueryTimingCollector::record($start, $sql); | ||
| } |
| /** | ||
| * Per-pattern bucket for finding N+1s during profiling. | ||
| * | ||
| * @var array<string, array{count:int, totalMs:float, sample:string}> | ||
| */ | ||
| public static array $patterns = []; | ||
|
|
||
| public static function record(float $startedAt, ?string $sql = null): void | ||
| { | ||
| self::$totalMs += (microtime(true) - $startedAt) * 1000.0; | ||
| $ms = (microtime(true) - $startedAt) * 1000.0; | ||
| self::$totalMs += $ms; | ||
| self::$count++; | ||
|
|
||
| if ($sql !== null) { | ||
| $pattern = self::normalize($sql); | ||
| if (!isset(self::$patterns[$pattern])) { | ||
| self::$patterns[$pattern] = ['count' => 0, 'totalMs' => 0.0, 'sample' => $sql]; | ||
| } | ||
| self::$patterns[$pattern]['count']++; | ||
| self::$patterns[$pattern]['totalMs'] += $ms; | ||
| } |
| // Temporary N+1 candidate logger (profiling-only — remove on cleanup). | ||
| if ($dbCount >= 20) { | ||
| foreach (\App\Http\Middleware\Doctrine\QueryTimingCollector::topPatterns(10) as $row) { | ||
| Log::warning('N+1 candidate', [ | ||
| 'count' => $row['count'], | ||
| 'totalMs' => $row['totalMs'], | ||
| 'sample' => mb_substr($row['sample'], 0, 240), | ||
| ]); | ||
| } | ||
| } |
| if (method_exists($attendee, 'getMember')) { | ||
| $m = $attendee->getMember(); | ||
| if ($m !== null && method_exists($m, 'getId') && $m->getId()) { | ||
| $memberIds[] = $m->getId(); |
0dbb393 to
ba81308
Compare
Summary
Same profiling-driven methodology as #549. Stacked on
hotfix/cache-optimizations.Full notes in
adr/003-attendees-endpoint-n-plus-1-elimination.md.What changed
Reusable infra:
ParametrizedGetAll::_getAllgains an optionalcallable $afterQuery = nullhook. Fires between data load and$response->toArray()— lets callers warm caches / batch-load related entities without touching the trait body. Backward-compatible.Targeted fixes:
Summit::getSpeakerByMemberper-instance memo +preloadSpeakersByMemberIdsbatch (3 lookups × N members → 3 batch queries with member fetch-join)SELECT a, n)SELECT a, t, b)SELECT a, tg)SELECT a, m)All five batch queries run from one closure passed to the trait's new
afterQueryparameter.Why total time only dropped 12% despite -57% queries
DB latency on the model database is ~25-30ms per query. We removed 47 queries' worth of latency but the remaining 36 still cost ~1000ms collectively. DB latency, not N+1 count, is now the dominant component — next investigation would be at the infrastructure layer (connection pool, replica targeting, query batching), not application code.
What was intentionally NOT fixed
PresentationSpeakerSELECT × 7 — deeper-chain access; ~6ms savingsCOUNT(MemberID)× 6 — non-current-user belongsToGroupSET TRANSACTION× 3 — connection lifecycleEach would save 10-50ms; diminishing returns.
Stacked on #549
Includes a merge commit from
hotfix/cache-optimizations(events PR). The traitafterQueryhook is technically introduced here but applies cleanly to both branches and will only flow into main once events PR merges first.