how to get text from an unclosed tag/selector?

[dynabler]

<item class="">
  <guid>0123456789</guid>
  <title>Lorem Ipsum is simply dummy text of the printing and typesetting industry.</title>
 <description>Lorem Ipsum is simply dummy text of the printing and typesetting industry.</description>
 <link>https://example.com/etc/um_innervation_painfully.dump/index/
 <pubdate>Tue, 12 Jan 1984 12:23:34 GMT</pubdate>
 <guid ispermalink="false">https://example.com/etc/0123456789/um_innervation_painfully.list/index"</guid>
 <media:thumbnail url="https://placehold.co/600x400" class="">
    <media:content medium="image" type="image/jpeg" url="https://placehold.co/600x400">
     <media:title></media:title>
     <media:description></media:description>
     <media:rating scheme="urn:simple">none</media:rating>
     <media:copyright url="/help/copyright/">© 1984-2024 Right Apple, Inc. No Rights Reserved</media:copyright>
     </media:content>
  </media:thumbnail>
</item>

I'm trying to scrape a page that consist of only <item></item>. I can get to everything, except for <link>, because that one is missing it's closing tag </link>. My code:

// flyscrape config setting
...
// scraping code
export default function({ doc, absoluteURL }) {
  const releases = doc.find("item"); 

  return {
    releases: releases.map((release) => {
      const pubdate = release.find("pubdate");
      const title = release.find("title");
      const desc = release.find("[type='image/jpeg']");

      return {
        title: title.text(),
        pubdate: pubdate.text(),
        desc: desc.attr("url"),
      };
    }),
  };
}

I have tried a couple of things, nodes, child selectors, etc, but no hail. Noteworthy mention is guid which is repeated in the <link>. Any ideas how I can get <link>? Thank you!

[philippta]

It seems like that the unclosed link element is treated as an empty element as in:

<link /> https://example.com/etc/um_innervation_painfully.dump/index/

This means that the link you are looking for is a child to the <item> element.

To still extract the link you could do the following:

1. Get all text contents of the item and split it by the new line character, which gives you a list of texts.
2. Find the correct item (or index) in the list, that contains the link you're looking for.

var link = release
    .text() // the all text contents
    .split("\n")[4] // split the text by new line character and select the 5th (zero based) line
    .trim() // remove the whitespace around that line
    
console.log(link) // https://example.com/etc/um_innervation_painfully.dump/index/

[dynabler]

@philippta

The usage of .split()[] as shown here in your last comment, prevents URL pagination from running #38 (or the other way around?)

If I add URL pagination #38 and use split()[] in doc.find or return {}, the script doesn't run.

Furthermore, it also prevents other modes of pagination, like follow, allowedURLs, depth etc. from running.

It does NOT affect single page url, depth 0 from running, only list pages and navigation is affected.

On your top of mind, any idea what might be causing this issue?

[philippta]

If the pagination stopped working I assume that there's a runtime error in the script. This can occur when you try to split a text into its likes but then try to access a particular line that doesn't exist (e.g. the text has 3 lines, but you try to access line number 4).

To guard against this what you can do is something like this:

var allLines = release.text().split("\n"); // Split the text into individual lines.

var foundLine;                             // Create a variable to store the found line.

if (allLines[4]) {                         // Ensure the line exists and is not empty.
    foundLine = allLines[4].trim();        // Grab the 5th (0-indexed) line and clean it.
} else {
    foundLine = "line not found";          // Line not found, we can write something in there or leave it empty.
}

return foundLine;                          // return or use the found line somewhere, e.g. console.log(foundLine)

[dynabler]

I have come to a realization about Flyscrape follow, depth, allowedURLs/blockedURLs, but it's a different topic, which I have put here: #56