Many feeds keep the latest N news items, so if the publisher posts frequently or we pause fetching for a long time, we can miss items. To help surface that to users, we should record time ranges where fetching was paused or where items may have been dropped, and use those gaps to indicate potential data loss.
A simple, practical approach is to inspect the return value of add_news(...):
|
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] |
|
pub struct AddingNewsOutput { |
|
/// Newly added news items or new versions of existing ones. |
|
pub new: Vec<Uuid>, |
|
/// Were seen again in the feed without any change or there is a newer version of them identified in `new`. |
|
pub touched: Vec<Uuid>, |
|
} |
If the method's touched set is empty for a given run, that implies none of the incoming items matched existing ones, so the interval between now and the LastFetcheAt value of the last inserted item may be treated as a blind gap.
Additionally, it would be reasonable to allow informants to explicitly declare gaps when they have information from the source that indicate it either explicitly or implicitly. This gives the system both automatic detection and manual set so we can reliably inform users when news may be missing.
This feature may also benefit the scheduler, so if gaps were frequently detected in a source, we may execute the informant more frequently.
Many feeds keep the latest
Nnews items, so if the publisher posts frequently or we pause fetching for a long time, we can miss items. To help surface that to users, we should record time ranges where fetching was paused or where items may have been dropped, and use those gaps to indicate potential data loss.A simple, practical approach is to inspect the return value of
add_news(...):aleym_core/src/db/impl_news_storage.rs
Lines 44 to 50 in a51dff9
If the method's touched set is empty for a given run, that implies none of the incoming items matched existing ones, so the interval between now and the
LastFetcheAtvalue of the last inserted item may be treated as a blind gap.Additionally, it would be reasonable to allow informants to explicitly declare gaps when they have information from the source that indicate it either explicitly or implicitly. This gives the system both automatic detection and manual set so we can reliably inform users when news may be missing.
This feature may also benefit the scheduler, so if gaps were frequently detected in a source, we may execute the informant more frequently.