Contradictions in 013 - Pathname Server Component Eval

[wsimmonds]

Hi,

First: Thanks for publishing these - I look forward to more of them and think they'll help to hugely improve the performance of LLMs in real world scenarios.

I've been experimenting with Claude Skills and attempting to improve the eval pass rates - with some success having got Claude Haiku 4.5 up to 50% after not much work.

In looking into failing evals, it looks like eval 013-pathname-server is not correct (in fact, no LLM passes on it according to https://nextjs.org/evals).

Expectations

Purpose from Readme:

This evaluation tests the ability to create a server component that uses URL pathname parameters to fetch data from an API, demonstrating dynamic routing and server-side data fetching patterns.

Expected result from Readme:

The implementation should create a dynamic route structure:

app/
├── product/
│   └── [id]/
│       └── page.tsx

The actual test case, however only looks for dynamic routes directly within the app/ folder itself - not within a nested folder like "product" or "products":

test('Dynamic route directory structure exists', () => {
  // Check for dynamic route like [id] or [productId]
  const appDir = join(process.cwd(), 'app');
  const entries = readdirSync(appDir, { withFileTypes: true });

  const hasDynamicRoute = entries.some(
    (entry: any) =>
      entry.isDirectory() &&
      entry.name.startsWith('[') &&
      entry.name.endsWith(']')
  );

  expect(hasDynamicRoute).toBe(true);
});

As from https://github.com/wsimmonds/next-evals-oss/blob/eaf7596897409837584fb94444c33773c18e05dd/evals/013-pathname-server/input/app/page.test.tsx#L5

Based upon the prompt:

Create a server component that uses the path name ID to fetch a product from an API. Use the Next.js App Router. Only output code for a single server component, not the API.

My personal take would be to expect it under app/products[id]/page.tsx but given singular/plural being convention/preference instead of absolute, it seems like the test case should be amended such that either would work. Though I caveat, I'm certainly not a Next expert!

Happy to make a PR to this effect if desired.

[mclenhard]

Good catch, we should either be explicit in the prompt or improve the tests to catch in nested folders. Checking nested folders seems like the better approach overall though.

Would love a PR! Let me know if I can be helpful in any way.