Improve llms.txt generation for API-heavy documentation

[network-charles]

A new enhancement was released to the generated llms.txt file; the page descriptions are truncated to 300 characters.
Source: https://www.mintlify.com/docs/ai/llmstxt#llms-txt

While I appreciate the new llms.txt character limit optimization process, the suggested workaround defeats the purpose of using a managed documentation platform. Many customers chose Mintlify so they could focus on documenting their product rather than maintaining documentation infrastructure. When the auto-generated llms.txt becomes unusable due to aggressive truncation, and the sudden alternate recommended solution is to override and curate the file manually, the maintenance burden shifts back to us as customers. At that point, the platform is no longer solving the problem it was intended to solve.

This also puts Mintlify customers who have API references in a difficult position, as API documentation naturally consumes a large portion of the available character limit.

A more scalable platform-level solution would be to automatically separate API references into their own generated LLM index (for example, llms-api.txt) while preserving a high-quality primary llms.txt for guides, concepts, and tutorials. However, the llms.txt references the custom llms-api.txt URL. This would allow customers to continue benefiting from automated generation without taking on the responsibility of manually maintaining LLM-specific documentation files.

Another scalable approach is to create a root llms.txt by high-level navigation. For example, the root llms.txt file points to the sub-roots of pages, such as Mintlify tabs or product navigation. The llms.txt generation script can handle this based on if-else conditions.

An example of an llms.txt file nested by navigation.

# MongoDB Documentation

> MongoDB is the leading document database. This index covers all MongoDB

> products, drivers, and tools documentation.

## Products

- [Atlas](https://www.mongodb.com/docs/atlas/llms.txt): MongoDB Atlas cloud database

- [Atlas CLI](https://www.mongodb.com/docs/atlas-cli/llms.txt): Command-line interface for Atlas

- [Compass](https://www.mongodb.com/docs/compass/llms.txt): GUI for MongoDB

- [MongoDB Server](https://www.mongodb.com/docs/manual/llms.txt): Server documentation

## Drivers

- [Python Driver](https://www.mongodb.com/docs/drivers/pymongo/llms.txt): PyMongo driver

- [Node.js Driver](https://www.mongodb.com/docs/drivers/node/llms.txt): Node.js driver

- [Java Driver](https://www.mongodb.com/docs/drivers/java/llms.txt): Java sync and reactive drivers

These suggested design decisions are based on the Agent documentation spec, specifically, the Progressive Disclosure for Large Documentation Sets.

https://agentdocsspec.com/spec/#progressive-disclosure-for-large-documentation-sets

These approaches provide better optionality.