Some news sources have a number-based indexing, where news items are ordered. If it allow us to do a fetch by index or index range, we can benefit from this to not loos items and to fetch exactly from where we stooped in the last fetch.
Also, when initially adding the source, we may start from the first available news item in the source rather than just starting by fetching the last 20 items (just an example number). This could be optional for the user to specify how far we should go.
Currently only the Telegram Web informant could support this, since all channel posts are indexed by number.
We could also specify the maximum number of items to be fetched in each scheduled operation. We may store this as part of the informant parameters, for example, in Telegram Web we get a maximum of 20 items in each fetch, so we could go to the next index range and fetch the next 20 without waiting for the next scheduled fetch.
We need to store an index for each in the database, and we shouldn't reuse the provided_id field for this. It's possible that some sources provide indexes that are ditched from the provided ID of a news item. We should properly expect that this could happen (e.g. news index is assigned for updated items while the provided ID is still the same).
Also, we need to pass the first and the last index to the informant when it is executed. Should we assume that all indexes are numbers and are sorted sequentially? No. Moreover, if we stored it as a string in the database we need a solution to find the minimum and maximum values.
Another solution would be to have an indexes table to store the minimum and maximum rather than trying to find them from the news items table.
Some news sources have a number-based indexing, where news items are ordered. If it allow us to do a fetch by index or index range, we can benefit from this to not loos items and to fetch exactly from where we stooped in the last fetch.
Also, when initially adding the source, we may start from the first available news item in the source rather than just starting by fetching the last 20 items (just an example number). This could be optional for the user to specify how far we should go.
Currently only the Telegram Web informant could support this, since all channel posts are indexed by number.
We could also specify the maximum number of items to be fetched in each scheduled operation. We may store this as part of the informant parameters, for example, in Telegram Web we get a maximum of 20 items in each fetch, so we could go to the next index range and fetch the next 20 without waiting for the next scheduled fetch.
We need to store an index for each in the database, and we shouldn't reuse the
provided_idfield for this. It's possible that some sources provide indexes that are ditched from the provided ID of a news item. We should properly expect that this could happen (e.g. news index is assigned for updated items while the provided ID is still the same).Also, we need to pass the first and the last index to the informant when it is executed. Should we assume that all indexes are numbers and are sorted sequentially? No. Moreover, if we stored it as a string in the database we need a solution to find the minimum and maximum values.
Another solution would be to have an indexes table to store the minimum and maximum rather than trying to find them from the news items table.