de_hypertext

Framework for parsing html into Rust structures.

Caution

The library is currently an experiment for my personal use. It's not at all ready for production.

Usage

cargo add de_hypertext

Example

use de_hypertext::Deserialize;
use de_hypertext::Deserializer;
use std::error::Error;

#[derive(Debug, Deserialize)]
struct BooksPage {
    #[de_hypertext(selector = "title", transform = |x: String| x.trim().to_string())]
    title: String,
    #[de_hypertext(selector = ".pager > .current", transform = |x: String| x.trim().to_string())]
    pages: String,
    #[de_hypertext(selector = ".row > li")]
    items: Vec<BookItem>,
}

#[derive(Debug, Deserialize)]
struct BookItem {
    #[de_hypertext(selector = "h3 > a", attribute = "href")]
    url: String,
    #[de_hypertext(selector = "h3 > a")]
    name: String,
    #[de_hypertext(selector = ".price_color")]
    price: String,
    #[de_hypertext(selector = ".star-rating", attribute = "class")]
    stars: String,
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let html = reqwest::get("https://books.toscrape.com/")
        .await?
        .text()
        .await?;
    let result = BooksPage::from_html(&html)?;
    println!("{result:#?}");
    Ok(())
}

Attributes

Attribute	Description
selector	CSS selector applied to the current element. Omit it to read the current element itself.
attribute	Read an HTML attribute instead of the element's text.
transform	Closure applied to the extracted String.

They combine freely:

#[de_hypertext(
    selector = ".price",
    attribute = "data-amount",
    transform = |x: String| x.replace("€", "")
)]
price: String,

Field types

Type	Behaviour
String	Text content, or the value of attribute. Missing element or attribute is an error.
Option<String>	Same, but a missing element or attribute yields None.
Vec<String>	Every element matching selector.
Vec<T>	Every match deserialized into T. Requires a selector.
T	A nested struct. Without a selector the current element is passed through.

#[derive(Debug, Deserialize)]
struct Page {
    #[de_hypertext(selector = ".subtitle")]
    subtitle: Option<String>,
    #[de_hypertext(selector = ".tag")]
    tags: Vec<String>,
    #[de_hypertext(selector = "a", attribute = "href")]
    links: Vec<String>,
    #[de_hypertext(selector = "footer")]
    footer: Footer,
}

Enums

Variants are tried in order, and the first one that deserializes wins. Use this when a page has several possible shapes.

#[derive(Debug, Deserialize)]
enum Pricing {
    Discounted(Discounted),
    Regular(Regular),
}

Each variant's field must itself implement Deserialize. Unit variants are not supported.

Errors

DeserializeError records where deserialization failed, including the path through nested structs and the index within a Vec. to_query_selector renders that path as a JavaScript expression you can paste into a browser console to see what the selector actually matched:

#[derive(Debug, Deserialize)]
struct Page {
    #[de_hypertext(selector = "footer")]
    footer: Footer,
}

#[derive(Debug, Deserialize)]
struct Footer {
    #[de_hypertext(selector = "span")]
    note: String,
}

// the footer is present, but the span inside it is not
let err = Page::from_html("<footer></footer>").unwrap_err();
println!("{:?}", err.to_query_selector());
// Some("document.querySelector('footer').querySelector('span')")