Description
To be really blunt, the documentation is of pretty low quality. I've just spent several hours trying to find how to apply a specific schema to the load_parquet() function, and it's as simple as just supplying it directly: load_parquet(schema=[Schema object]). I'm glad the solution is simple, but to spend this long trying to find a solution is indicative of a bigger issue: the documentation fails to mention that this is possible, nor what other parameters a Dataset object might take. Navigating to the actual Dataset page is difficult, since the search does not return meaningful results, nor are mentions of it hyperlinked, only highlighted.
Take this page on the read_parquet method. The listed parameters aren't exactly described or given examples, their descriptions are already clear from the name: "filesystem - The filesystem implementation to read from." This information is obvious from the name, but what types of values are accepted? I will assume string values, but what strings, exactly?
I'm sure this is a great library, I can tell there's an enormous amount of work and effort that has gone into it. But to ship with such uninformative, inaccessible and poorly structured documentation makes it incredibly frustrating for new people like me to get to grips with it. I'd love to use this library further but if anything more complex than just loading in data is equally unclear, I won't be able to...
The solution I found was from this stackoverflow post, which I only found by googling the uninformative error message I received when calling .repartition(): pyarrow.lib.check_statuspyarrow.lib.ArrowNotImplementedError: Unsupported cast from double to null using function cast_null. Perhaps such an error message should show a snippet of the record and/or column(s) in question, or mention needing to verify no null entries exist in the dataset's schema?
Link
https://docs.ray.io/en/latest/data/consuming-datasets.html#consuming-datasets
https://docs.ray.io/en/latest/data/api/input_output.html#ray.data.read_parquet
Description
To be really blunt, the documentation is of pretty low quality. I've just spent several hours trying to find how to apply a specific schema to the
load_parquet()function, and it's as simple as just supplying it directly:load_parquet(schema=[Schema object]). I'm glad the solution is simple, but to spend this long trying to find a solution is indicative of a bigger issue: the documentation fails to mention that this is possible, nor what other parameters a Dataset object might take. Navigating to the actual Dataset page is difficult, since the search does not return meaningful results, nor are mentions of it hyperlinked, only highlighted.Take this page on the
read_parquetmethod. The listed parameters aren't exactly described or given examples, their descriptions are already clear from the name: "filesystem - The filesystem implementation to read from." This information is obvious from the name, but what types of values are accepted? I will assume string values, but what strings, exactly?I'm sure this is a great library, I can tell there's an enormous amount of work and effort that has gone into it. But to ship with such uninformative, inaccessible and poorly structured documentation makes it incredibly frustrating for new people like me to get to grips with it. I'd love to use this library further but if anything more complex than just loading in data is equally unclear, I won't be able to...
The solution I found was from this stackoverflow post, which I only found by googling the uninformative error message I received when calling
.repartition():pyarrow.lib.check_statuspyarrow.lib.ArrowNotImplementedError: Unsupported cast from double to null using function cast_null. Perhaps such an error message should show a snippet of the record and/or column(s) in question, or mention needing to verify no null entries exist in the dataset's schema?Link
https://docs.ray.io/en/latest/data/consuming-datasets.html#consuming-datasets
https://docs.ray.io/en/latest/data/api/input_output.html#ray.data.read_parquet