Add 'recently indexed stories' table to source page in directory

[pgulley]

We currently only show 'recently discovered' stories from the RSS fetcher in the collection page. We have the spare bandwidth to just execute a story-list call against the index itself, which tells a more meaningful story to the user. The 'recently discovered' table should be moved to a new tab.

[philbudne]

I've looked into this, and have these observations:

1. The existing SampleStories component

   web-search/mcweb/frontend/src/features/search/results/SampleStories.jsx

   Line 24 in 2f62d09

   export default function SampleStories() {

   displays decscriptive text that is explicitly query related, and possibly oriented towards multiple user queries

2. The list of rss-fetcher discovered stories is ordered by how recently the stories were fetched, and there is currently no way to specify any ordering by EITHER the ES indexed_date or publication_date fields, BUT ES does support ordering, so the providers library random_sample call could be extended to allow ordering (to select the most recently indexed stories) OR an order_by argument could be implemented at the _basic_search level if it's determined to be useful in other contexts.

NOTE: I already have extended the ES provider _basic_search core method to allow the start/end date to be interpreted as indexed date instead of published date if the kwarg indexed_date is truthy.

[philbudne]

On number 2 above, I may have been barking up the wrong tree, and the providers paged_items call already HAS sort_field and sort_order arguments, and ALL the NEWEST entries is what we're displaying for RSS fetcher stories, so it's likely no providers changes are needed!

The story_list endpoint already defaults to sorting by indexed_date and sort_order should be accepted as an optional parameter to the story_list endpoint, and passed thru to paged_items. The only missing bit may be being able to specify the indexed_date boolean (which may only require that it be added to _MEDIA_CLOUD_EXTRA_PROPS or _MEDIA_CLOUD_INT_PROPS) in backend/search/utils.py. The rss fetcher keeps story rows for 90 days, so the "proper" search range would be the last 90 days of indexed stories...

[philbudne]

And as for number one, a new component <RecentStories> could be used to encapsulate the fetch/display....

[philbudne]

So:

1. no changes needed for sort_field or _order
2. furthermore: completely removing pub_date from the query would pull in stories with no pub_date!!
3. maybe use start_date of EARLIEST_AVAILABLE_DATE and end_date tomorrow, PLUS a query string with indexed_date:[90_days_ago TO *]??

[pgulley]

That sounds like a plan to me!

[philbudne]

Yeah, in the end the actual query is simple, and runs quickly: start_date: earliest searchable pub date end_date: newest accepted pub date (+90 days??) query: indexed_date:[90_days_ago TO *] But now I'm lost in a maze of twisty UI Components!! The search works equally well for BOTH sources AND collections, so it's tempting to make it visible for both (except that displaying results for a collection means including the source/domain name (and icon!)). It wouldn't fit side by side in the collection screen, so a tab seems like the natural choice....