Is it easy to support alternative domains on child sources?

[rahulbot]

For examples such as svz.de redirecting to a child source of nordkurier.de it'd be nice if we could also make the former an "alternative domain" of the latter. The UI and perhaps other code prevents this. Is it hard to change it?

The menu item could be enabled by removing this line:

web-search/mcweb/frontend/src/features/sources/util/AdvancedMenu.jsx

Line 292 in 4bbece3

disabled={!!source.url_search_string}

[philbudne]

There are a number of subissues, I'll try to enumerate them. Most seem trivial, But I'd be surprised if this is an exhaustive list! 1. Backend changes a. The AlternativeDomain object would need a url_search_string member. The Source url_search_string is defined as: url_search_string = models.CharField(max_length=1000, blank=True, null=True) Like the Source field, we would want it to either be NULL or a non-empty string (the blank argument is for input validation?) ISTR this took a while to get to the point where Source.url_search_string was never set to the empty string, so looking at Source handling might be helpful? b. AlternativeDomainSerializer needs the new field. Maybe this class is the place to check for empty string in url_search_string and cooerce to NULL??? c. SourcesViewSerializer.get_alternative_domains needs to return the url_search_string field? I can't find any calls to that method (might be an unused vestage??) d. honor the url_search_string if set in generating provider arguments in mcweb.search.utils._for_media_clould This is the one non-trivial (or at least the least trivial) task, however it's only dependency is (a) above. As written, the code assumes that there is at most one (domain, url_search_string) pair per source id, which would no longer be true. Querying the AlternativeDomain table needs to be done earlier (and looking at the code, I don't think it's handling alt domains for sources in collections properly right now!!!!). I suspect the code might be simpler if a UNION query for sources and sources in collections was done and get the database to only return distinct sources (which I think may be an inherent property of UNION queries). It might even be possible to wrap the alternativeDomain table query into a single UNION query that returns just (domain, url_search_string) pairs. A great deal of the gymnastics in the code is to handle duplicate sources, and I think I can make that go away. I'd like to take a crack at this. I would start with creating a monitor.py admin command that takes source and collection ids on the command line so that the code can be debugged/tested/timed in vitro, including an option to enable output of the generated/executed SQL! This is also the place to validate source and collection ids, which I becomes important now that we're starting to create alternative domains (deleting old source ids in the process). I think it might be possible with only one additional query, something like: SELECT COUNT(SELECT id from sources_source WHERE id IN (user_sources_list)) as sources_found, COUNT(SELECT id from sources_collection WHERE id IN (user_collections_list)) as collections_found; 2. Front end changes a. Anything that displays alternate domains needs to display the url_search_string if it's set. b. The above check to disable conversion to an alternative domain if it has a url_search_string needs to be removed. c. Optionally allow adding Alternative entries to an existing source. ie; add a new field to the Add an Alternative Domain for NAME, this should be the canonical domain and should not be a source already. dialog reached from the Source > ADVANCED... > CREATE ALTERNATIVE ... menu item. This may expose back end code paths that are missing!

[rahulbot]

Thanks for thinking it though. I have a question about an early point: why would AlternativeDomain object need a url_search_string? The alternative domain is on the child source, so what if we assume that the url_search_string applies to the source domain and not any other alternative ones. That specifically solves the "source turned into a sub-path on another domain" situation and not others, but doesn't it simplify a number of the domino changed you described above.

[philbudne]

Sorry, I didn't look closely enough at the example. I assumed the task was the "hard one" that I volunteered to look into at the meeting! I'll cop the excuse that it was barely past noon, and my brain kicked in! I don't think there is anything other than the UI check you spotted that's in the way of adding a simple/parent domain as an alternative to a child source. The one caveat is my discovery that as I just discovered, alternative domains from sources searched via collections are NOT being checked (ie; basic breakage how alternative domains were added to search). It's a two step process no matter how you do it, either: Change existing source name and url_search_string, then "add alternative domain" for the new, simple domain. or create new child source and convert the existing/old simple/parent source into an alternative. I can't say whether either path should or does require that the parent domain for the new, primary (child) source exist. I'm pretty sure your change to bbn.co.uk left some orphan children this morning, but ISTR seeing that all of them (srcs 18204, 39602, 40830 and 41699) had "?" displayed for "NEW STORIES IN", meaning the weekly stats update hadn't found any stories with pub_dates that matched the existing search string. BUT at least for 41699 there must be a working feed associated with the source(*), because "Latest Stories" are shown with URLs like www.bbc.com/russian/ while the search string has www.bbn.co.uk/russian(**) (*) Despite the catechiem that "child sources don't have feeds", this one does (something to add as a wish list item for a full directory health check?): mcweb_db=# select id, url from sources_feed where source_id = 41699; id | url --------+---------------------------------------- 89213 | http://www.bbc.co.uk/russian/index.xml 373751 | http://www.bbc.com/russian/index.xml (2 rows) (**) Entering url:https\://www.bbc.com/russian/* does find recent content, but trying larger date ranges causes a spinner on the screen and grinding noises in my head. A large date range query failed, but was able to verify that there is content back to 4/2025. Finally, I can easily imagine someone will someday think that adding a simple domain as an alternative on a child source will somehow, magically look at the full URL (inherited from the existing url_search_string). BUT I'm not sure that adding language saying "I understand that this captures all stories indexed under the alternative canonical domain" would reduce confusion without an example like local-paper.com being turned into conglomerate.com/locality Related (because we're talking about child sources), I'm not sure how many people can successfully create working child sources, whether would be helpful to have some way for source editors to test whether a url_search_string matches (some) of what is intended (as they're creating a source?), since it's easy to enter a useless url_search_string (one that doesn't exactly match the full (non-canonical) domain in the story URLs).