Describe the bug
I am trying to use cuml.feature_extraction.text.TfidfVectorizer with char_wb analyzer and ngrams, but .fit_transform() fails.
Steps/Code to reproduce bug
Consider this simple example which compares TfidfVectorizer from cuml and from scikit-learn using identical setup:
import pandas as pd
from cuml.feature_extraction.text import TfidfVectorizer as cuml_TfidfVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer as sk_TfidfVectorizer
ngram_range = (2, 6)
tfidf_vectorizer_cuml = cuml_TfidfVectorizer(analyzer='char_wb', ngram_range=ngram_range)
tfidf_vectorizer_sk = sk_TfidfVectorizer(analyzer='char_wb', ngram_range=ngram_range)
texts = [
"Hi there, nice to meet you",
"How are you?",
]
texts = pd.Series(texts)
features_sk = tfidf_vectorizer_sk.fit_transform(texts)
print(features_sk.shape)
features_cuml = tfidf_vectorizer_cuml.fit_transform(texts)
print(features_cuml.shape)Expected behavior
features_cuml is created and is a sparse matrix with the TF-IDF features of the analyzed input texts.
In reality, running this results in the following output:
- The shape of
features_sk is correct:
- Trying to
.fit_transform() the tfidf_vectorizer_cuml yields ValueError: cannot reindex on an axis with duplicate labels with this traceback:
Traceback (most recent call last):
File "/home/nikolay/Projects/cuml_exploration/cuml_play_issue_mwe.py", line 16, in <module>
features_cuml = tfidf_vectorizer_cuml.fit_transform(texts)
File "/home/nikolay/Projects/cuml_exploration/.venv/lib/python3.13/site-packages/cuml/feature_extraction/_tfidf_vectorizer.py", line 262, in fit_transform
X = super().fit_transform(raw_documents)
File "/home/nikolay/Projects/cuml_exploration/.venv/lib/python3.13/site-packages/cuml/feature_extraction/_vectorizers.py", line 596, in fit_transform
tokenized_df = self._create_tokenized_df(docs)
File "/home/nikolay/Projects/cuml_exploration/.venv/lib/python3.13/site-packages/cuml/feature_extraction/_vectorizers.py", line 245, in _create_tokenized_df
self.get_ngrams(docs, n, doc_id) for n in range(min_n, max_n + 1)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/home/nikolay/Projects/cuml_exploration/.venv/lib/python3.13/site-packages/cuml/feature_extraction/_vectorizers.py", line 218, in get_ngrams
ngram_sr, ngram_count, token_count_sr = self.get_char_ngrams(
~~~~~~~~~~~~~~~~~~~~^
ngram_size, str_series, doc_id_sr
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/nikolay/Projects/cuml_exploration/.venv/lib/python3.13/site-packages/cuml/feature_extraction/_vectorizers.py", line 161, in get_char_ngrams
tokens = tokens.str.cat(padding)
File "/home/nikolay/Projects/cuml_exploration/.venv/lib/python3.13/site-packages/cudf/core/accessors/string.py", line 359, in cat
others = others.reindex(parent_index)
File "/home/nikolay/Projects/cuml_exploration/.venv/lib/python3.13/site-packages/cudf/core/series.py", line 911, in reindex
series = self._reindex(
deep=copy,
...<8 lines>...
tolerance=tolerance,
)
File "/home/nikolay/Projects/cuml_exploration/.venv/lib/python3.13/site-packages/cudf/core/indexed_frame.py", line 3930, in _reindex
raise ValueError(
"cannot reindex on an axis with duplicate labels"
)
ValueError: cannot reindex on an axis with duplicate labels
Process finished with exit code 1Environment details (please complete the following information):
- Environment location: Bare-metal
- Linux Distro/Architecture: Ubuntu 24.04.4 LTS
- GPU Model/Driver: NVIDIA RTX A2000 Laptop GPU / Driver Version: 580.173.02
- CUDA: 13.0
- Method of cuDF & cuML install:
uv pip install
Output of uv pip list
Package Version Editable project location
------------------------------------------------- --------------- ------------------------------------------------------
accelerate 1.14.0
aiofiles 25.1.0
aiohappyeyeballs 2.7.1
aiohttp 3.14.3
aiolimiter 1.2.1
aiosignal 1.4.0
annotated-doc 0.0.4
annotated-types 0.7.0
anthropic 0.119.0
anyio 4.13.0
arpeggio 2.0.3
asttokens 3.0.1
attrs 26.1.0
av 18.0.0
baml-py 0.223.0
beautifulsoup4 4.14.3
bidict 0.23.1
blingfire 0.1.8
blinker 1.9.0
cachebox 5.2.3
cachetools 7.1.6
catalogue 2.0.10
certifi 2026.5.20
cffi 2.0.0
charset-normalizer 3.4.7
click 8.4.2
cloudpickle 3.1.2
cmdstanpy 1.3.0
colorama 0.4.6
comm 0.2.3
contourpy 1.3.3
cryptography 48.0.0
cssselect 1.4.0
cucim-cu13 26.6.0
cuda-bindings 13.2.0
cuda-core 0.7.0
cuda-pathfinder 1.6.0
cuda-python 13.2.0
cuda-toolkit 13.3.1
cudf-cu13 26.6.0
cugraph-cu13 26.6.0
cuml-cu13 26.6.0
cupy-cuda13x 14.1.1
cuvs-cu13 26.6.0
cycler 0.12.1
dash 4.4.1
dash-ag-grid 35.3.0
dash-bootstrap-components 2.0.4
dask 2026.1.1
dask-cuda 26.6.0
dask-cudf-cu13 26.6.0
dask-expr 2.0.0
datasets 5.0.0
dateparser 1.4.1
debugpy 1.8.21
decorator 5.3.1
deepdiff 9.1.0
defusedxml 0.7.1
dill 0.4.1
diskcache 5.6.3
distributed 2026.1.1
distributed-ucxx-cu13 0.50.1
distro 1.9.0
dnn-cool 0.4.0
docker 7.2.0
docstring-parser 0.18.0
dotenv 0.9.9
elastic-transport 9.4.2
elasticsearch 9.4.1
emoji 2.15.0
eval-type-backport 0.4.0
executing 2.2.1
faker 40.35.0
fastcore 1.14.5
filelock 3.32.0
flagembedding 1.4.0
flask 3.1.3
flask-cors 6.0.5
flask-httpauth 4.8.1
flatbuffers 25.12.19
fonttools 4.63.0
frozenlist 1.8.0
fsspec 2026.7.0
geonamescache 3.0.1
gitdb 4.0.12
gitpython 3.1.50
google-api-core 2.33.0
google-auth 2.53.0
google-cloud-aiplatform 1.162.0
google-cloud-bigquery 3.42.2
google-cloud-core 2.6.0
google-cloud-resource-manager 1.18.0
google-cloud-speech 2.40.0
google-cloud-storage 3.13.0
google-cloud-texttospeech 2.37.0
google-cloud-translate 3.27.0
google-crc32c 1.8.0
google-genai 2.8.0
google-resumable-media 2.10.0
googleapis-common-protos 1.75.0
greenlet 3.5.4
grpc-google-iam-v1 0.14.4
grpcio 1.83.0
grpcio-status 1.83.0
gunicorn 26.0.0
h11 0.16.0
hf-xet 1.5.2
holidays 0.101
html2text 2025.4.15
httpcore 1.0.9
httpx 0.28.1
huggingface-hub 0.36.2
idna 3.18
imageio 2.37.4
importlib-metadata 8.7.1
importlib-resources 7.1.0
interchange 2021.0.4
ipykernel 7.2.0
ipython 9.14.0
ipython-pygments-lexers 1.1.1
ir-datasets 0.6.3
itsdangerous 2.2.0
janus 2.0.0
jedi 0.20.0
jinja2 3.1.6
jiter 0.16.0
joblib 1.5.3
json-repair 0.59.10
jsonpatch 1.33
jsonpointer 3.1.1
jupyter-client 8.8.0
jupyter-core 5.9.1
kaleido 0.2.1
kiwisolver 1.5.0
langchain-core 1.5.1
langchain-protocol 0.0.18
langchain-text-splitters 1.1.2
langsmith 0.10.10
lazy-loader 0.5
libcudf-cu13 26.6.0
libcugraph-cu13 26.6.0
libcuml-cu13 26.6.0
libcuvs-cu13 26.6.0
libkvikio-cu13 26.6.0
libnvforest-cu13 26.6.0
libraft-cu13 26.6.0
librmm-cu13 26.6.0
libucx-cu13 1.19.0
libucxx-cu13 0.50.0
lingua-language-detector 2.2.0
livekit 1.1.8
livekit-agents 1.5.17
livekit-api 1.2.0
livekit-blingfire 1.1.0
livekit-plugins-elevenlabs 1.5.17
livekit-plugins-google 1.5.17
livekit-plugins-silero 1.5.17
livekit-protocol 1.1.21
llvmlite 0.46.0
locket 1.0.0
lxml 6.1.1
lz4 4.4.5
markdown-it-py 4.2.0
markupsafe 3.0.3
matplotlib 3.11.1
matplotlib-inline 0.2.2
mdurl 0.1.2
ml-dtypes 0.5.4
mmap-ninja 0.9.0
mmap-ninja-dataframe 0.8.0
monotonic 1.6
mpmath 1.3.0
msgpack 1.2.1
multidict 6.7.1
multiprocess 0.70.19
narwhals 2.24.0
nest-asyncio 1.6.0
networkx 3.6.1
nltk 3.10.0
numba 0.64.0
numba-cuda 0.28.2
numpy 2.4.6
nvforest-cu13 26.6.0
nvidia-cublas 13.6.0.2
nvidia-cuda-cccl 13.3.3.4.1
nvidia-cuda-crt 13.3.73
nvidia-cuda-cupti 13.0.85
nvidia-cuda-nvcc 13.3.73
nvidia-cuda-nvrtc 13.3.33
nvidia-cuda-runtime 13.3.29
nvidia-cudnn-cu13 9.20.0.48
nvidia-cufft 12.3.0.29
nvidia-cufile 1.15.1.6
nvidia-curand 10.4.3.29
nvidia-cusolver 12.2.6.9
nvidia-cusparse 12.8.2.51
nvidia-cusparselt-cu13 0.8.1
nvidia-libnvcomp-cu13 5.2.0.13
nvidia-ml-py 13.610.43
nvidia-nccl-cu13 2.30.7
nvidia-nvimgcodec-cu13 0.8.0.22
nvidia-nvjitlink 13.3.33
nvidia-nvshmem-cu13 3.4.5
nvidia-nvtx 13.0.85
nvidia-nvvm 13.3.73
nvtx 0.2.15
nx-cugraph-cu13 26.6.0
onnx 1.22.0
onnxruntime 1.27.0
onnxruntime-gpu 1.27.0
openai 2.48.0
opentelemetry-api 1.39.1
opentelemetry-distro 0.60b1
opentelemetry-exporter-otlp 1.39.1
opentelemetry-exporter-otlp-proto-common 1.39.1
opentelemetry-exporter-otlp-proto-grpc 1.39.1
opentelemetry-exporter-otlp-proto-http 1.39.1
opentelemetry-instrumentation 0.60b1
opentelemetry-instrumentation-asyncio 0.60b1
opentelemetry-instrumentation-flask 0.60b1
opentelemetry-instrumentation-google-generativeai 0.61.0
opentelemetry-instrumentation-logging 0.60b1
opentelemetry-instrumentation-requests 0.60b1
opentelemetry-instrumentation-wsgi 0.60b1
opentelemetry-proto 1.39.1
opentelemetry-sdk 1.39.1
opentelemetry-semantic-conventions 0.60b1
opentelemetry-semantic-conventions-ai 0.5.1
opentelemetry-util-http 0.60b1
optimum 2.1.0
optimum-onnx 0.1.0
ordered-set 4.1.0
orderly-set 5.5.0
orjson 3.11.9
outcome 1.3.0.post0
packaging 26.2
pandas 2.3.3
pansi 2024.11.0
parso 0.8.7
partd 1.4.2
peft 0.19.1
pexpect 4.9.0
pgvector 0.5.0
phonenumbers 9.0.34
pickleshare 0.7.5
pika 1.4.2
pillow 12.3.0
platformdirs 4.10.0
plotly 6.9.0
polars 1.43.0
polars-runtime-32 1.43.0
prometheus-client 0.25.0
prompt-toolkit 3.0.52
propcache 0.5.2
prophet 1.3.0
proto-plus 1.28.2
protobuf 6.33.6
psutil 7.2.2
psycopg 3.3.4
ptyprocess 0.7.0
pure-eval 0.2.3
py2neo 2021.2.4
pyarrow 23.0.1
pyarrow-hotfix 0.7
pyasn1 0.6.3
pyasn1-modules 0.4.2
pycparser 3.0
pydantic 2.13.4
pydantic-core 2.46.4
pygments 2.20.0
pyjwt 2.13.0
pylibcudf-cu13 26.6.0
pylibcugraph-cu13 26.6.0
pylibraft-cu13 26.6.0
pymupdf 1.28.0
pyopenssl 26.2.0
pyparsing 3.3.2
pysocks 1.7.1
python-dateutil 2.9.0.post0
python-dotenv 1.2.2
python-stdnum 2.2
pytz 2026.3.post1
pyyaml 6.0.3
pyzmq 27.1.0
raft-dask-cu13 26.6.0
rapidfuzz 3.14.5
rapids-dask-dependency 26.6.0
rapids-logger 0.2.3
redis 8.0.1
regex 2026.7.19
requests 2.34.2
requests-file 3.0.1
requests-toolbelt 1.0.0
retrying 1.4.2
rich 15.0.0
rmm-cu13 26.6.0
safetensors 0.8.0
scikit-image 0.26.0
scikit-learn 1.9.0
scipy 1.18.0
scrubadub 2.0.1
selenium 4.46.0
sentence-transformers 5.6.1
sentencepiece 0.2.2
setuptools 82.0.1
shellingham 1.5.4
shortuuid 1.0.13
six 1.17.0
smmap 5.0.3
sniffio 1.3.1
sortedcontainers 2.4.0
sounddevice 0.5.5
soupsieve 2.8.4
sqlalchemy 2.0.51
sqlitedict 2.1.0
sqlmodel 0.0.39
stack-data 0.6.3
stanio 0.5.1
stanza 1.14.0
sympy 1.14.0
tabulate 0.10.0
tblib 3.2.2
tenacity 9.1.4
termcolor 3.3.0
textblob 0.15.3
textx 4.4.0
threadpoolctl 3.6.0
tifffile 2026.7.14
tiktoken 0.13.0
tldextract 5.3.1
tokenizers 0.22.2
toolz 1.1.0
torch 2.13.0
tornado 6.5.7
tqdm 4.69.0
traitlets 5.15.1
transformers 4.57.6
treelib 1.8.0
treelite 4.7.0
trio 0.33.0
trio-websocket 0.12.2
triton 3.7.1
typer 0.26.7
types-protobuf 7.34.1.20260518
typing-extensions 4.15.0
typing-inspection 0.4.2
tzdata 2026.3
tzlocal 5.4.4
ucxx-cu13 0.50.0
udapi 0.5.2
udtools 0.2.8
unoserver 3.7
urllib3 2.7.0
uuid-utils 0.17.0
voyageai 0.5.0
watchfiles 1.2.0
wcwidth 0.7.0
websocket-client 1.9.0
websockets 16.0
werkzeug 3.1.8
wrapt 1.17.3
wsproto 1.3.2
xxhash 3.8.1
yarl 1.24.5
yattag 1.16.1
zict 3.0.0
zipp 4.1.0
zstandard 0.25.0
Additional context
Nothing as of now.
Describe the bug
I am trying to use
cuml.feature_extraction.text.TfidfVectorizerwithchar_wbanalyzer and ngrams, but.fit_transform()fails.Steps/Code to reproduce bug
Consider this simple example which compares TfidfVectorizer from
cumland fromscikit-learnusing identical setup:Expected behavior
features_cumlis created and is a sparse matrix with the TF-IDF features of the analyzed input texts.In reality, running this results in the following output:
features_skis correct:.fit_transform()thetfidf_vectorizer_cumlyieldsValueError: cannot reindex on an axis with duplicate labelswith this traceback:Environment details (please complete the following information):
uv pip installOutput of uv pip list
Additional context
Nothing as of now.