Integrate `hf datasets` as a data source

[ParagEkbote]

Is your feature request related to a problem? Please describe.

The Hugging Face Hub recently surpassed 1M+ datasets. Many users store and share hf datasets on the Hub and apply it as a standard interface for accessing them. However, great_expectations does not currently provide Hugging Face datasets as a native datasource.

Describe the solution you'd like
Add support for Hugging Face datasets as a datasource in great_expectations. This would allow users to load datasets directly from the Hugging Face Hub and apply expectations to their data.

Describe alternatives you've considered
N.A.

Additional context
I would be interesting in contributing this integration, having done a similar integration contribution to dagster.

[ParagEkbote]

What do you think about the proposed integration?

cc: @joshua-stauffer

[joshua-stauffer]

Hi @ParagEkbote — thanks for reaching out and for offering to contribute this. A native Hugging Face integration could definitely be valuable for teams working with training, evaluation, and other AI/ML datasets.

Adding a new data source is an ongoing maintenance commitment for GX, so before we take it on I'd like two things in place:

• Community interest: I'd like to hear from other GX users who would use this integration, including the datasets and validation workflows they have in mind.
• A concrete design proposal (RFC): We need to agree on the scope and architecture before implementation begins.

I'll convert this issue to a discussion so the conversation stays open and others can weigh in.

The questions I'd most want an RFC to answer:

• What does a native data source give users over the already-working path of load_dataset(...).to_pandas() into a plain GX Pandas Data Source?
• How would Hugging Face's repository → configuration/subset → split → shard structure (with revision as a versioning axis across the repo) map onto GX's Data Source → Data Asset → Batch Definition → Batch model?
• Which Execution Engine would process the data? HF is a dataset registry and loading layer, not a compute backend; GX computes Metrics on Pandas, Spark, or SQL engines, so Pandas is the natural first target.
• What dataset types would the first version support? GX support for non-scalar data types is limited.
• How would the integration handle configurations, splits, pinned revisions, authentication, caching, and large datasets?
• Would streaming datasets (IterableDataset) be in or out of scope for v1?

Another lighter-weight approach would be a HF dataset docs recipe, similar to our existing Validate unstructured data tutorial.

Happy to keep exploring this with you through the discussion and RFC.