This is a Fork of Sds-Core package of Sds Project.
Sds Core is a library that defines core data types and transformations in Sds.
To use Sds Core, simply install sds-core from your package manager, e.g. pip:
pip install sds-coreTo develop for Sds Core, you need Python 3.10 through 3.14 and the uv package. You can then install it from your local clone's root directory:
uv sync --all-extrasTo run the pytest suite, execute:
uv run pytest -s test
Sds Core provides the foundational SdsDocument data model and API, as well as additional APIs for tasks like serialization and chunking, which are key to developing generative AI applications using Sds.
Sds Core defines the SdsDocument as a Pydantic model, allowing for advanced data model control, customizability, and interoperability.
In addition to specifying the schema, it provides a handy API for building documents, as well as for basic operations, e.g. exporting to various formats, like Markdown, HTML, and others.
👉 More details:
Different users can have varying requirements when it comes to serialization. To address this, the Serialization API introduces a design that allows easy extension, while providing feature-rich built-in implementations (on which the respective SdsDocument helpers are actually based).
👉 More details:
Similarly to above, the Chunking API provides built-in chunking capabilities as well as a design that enables easy extension, this way tackling customization requirements of different use cases.
👉 More details:
Please read Contributing to Sds Core for details.
If you use Sds Core in your projects, please consider citing the following:
@techreport{Sds,
author = "Deep Search Team",
month = 8,
title = "Sds Technical Report",
url = "https://arxiv.org/abs/2408.09869",
eprint = "2408.09869",
doi = "10.48550/arXiv.2408.09869",
version = "1.0.0",
year = 2024
}The Sds Core codebase is under MIT license. For individual model usage, please refer to the model licenses found in the original packages.