Download attached files in the Browser's context

[reox]

I often have the problem, that articles that are imported via the browser extension have linked PDFs, which I can not download via JabRef itself (for example via JabRef/jabref#5662 or by browsing to the PDF and then starting the browser extension). That might be because you have to be logged in into a website or due to other reasons. Some pages return an error 403 (sciencedirect, researchgate), others return 503 (wiley - maybe that one is a different problem though).

However, if I open the link in the browser, I get the PDF immediately.
Thus, it would be convenient to be able to download the PDF directly in the JabRef extension (so that it has the browser context and cookies (is that possible?)) and then send the downloaded PDF and metadata into Jabref.

edit: I quickly checked what's going on with wiley. The problem seems to be cloudflare. When I use wget or curl to access the PDF, I get immediately a 503 error and a html page which says at the bottom:

DDoS protection by <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing/" target="_blank">Cloudflare</a>

[tobiasdiez]

That's a good idea, and would be possible to implement using https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/downloads/download. We should also add an option that enables this in-the-browser download behavior, as some people may not want to download files at all. PR is very welcome.

[ror3d]

I've been thinking a bit on how this could be solved. Several points come to mind:

• As opposed to Zotero, JabRef's plugin doesn't directly add entries to the library, but sends them to be imported through the import dialog, because the communication between the two applications (jabref and plugin) is minimal. This makes asynchronous downloading of the attached files more complicated. Either:
  • the files have to be downloaded prior to opening JabRef so it already has the file information, or
  • some way needs to be devised to send the updated information to JabRef
    • then the possibility of adding a file but cancelling the import before it is downloaded and sent to JabRef, and other similar edge cases, have to also be considered.
• Having a more direct connection between the plugin and JabRef using connectNative sockets instead of sendNativeMessage might allow for easier control of these cases (the entry could be directly created and then the files attached to it once they are downloaded). This would require a bunch of work in different ways than the previous solution.
• Trying to download files from JabRef after-the-fact would still have the same issues with not having a proper context to download past paywalls.
  • A more direct connection through connectNative might allow for downloading files after-the-fact through paywalls more easily from inside JabRef
  • An embedded browser in JabRef could be implemented instead to be able to access such files. This could solve both problems through a different approach. It might be quite tricky to implement so it works well.

[tobiasdiez]

* Having a more direct connection between the plugin and JabRef using connectNative sockets instead of sendNativeMessage might allow for easier control of these cases (the entry could be directly created and then the files attached to it once they are downloaded). This would require a bunch of work in different ways than the previous solution.

This would be my preference. Either using connectNative or using a websocket connection (i.e. implement a websocket server in the main jabref application, similar in spirit to https://github.com/karlicoss/grasp/blob/master/extension/src/js/background.js). Not sure what is easier to implement and more stable. I sadly don't have the time to work on it myself, but would of course accept PRs and help with the implementation.

Such a connection would be also handy to implement other features. For example, #32.

[LuminousMatter]

Hi, I'm not sure if my download issue/experience exactly fits this discussion: the links to the actual PDF differ from the link grabbed by the extension. I suspect the platforms design is the reason, so the following Example is with one of my common literature plattforms ( https://www.sciencedirect.com/, https://www.elsevier.com/).

Example of my way to get a JabRef Entry setup correctly

• using bookworm, Firefox ESR & manually updated JabRef 5.15 (as I'm only getting started on understanding software details I do prefer the reliability of stable versions. Yes, i'm aware possible compatibility may be my actual issue here.)

1. the plugin sends meta data to JabRef correctly

2. the import selection have the download request
   

3. the imported entry is correct, except for the wrong file link (I expected the location of the downloaded file)

4. the actual PDF file has a different link than noted in the file-field.
   

5. Here only a manual selection & download get's the PDF.

6. If saved correctly: attaching the file with F7 'automatically set file links' works great & fast.

My conclusion: Something like a websocket could help deal with the style of different research platforms. Maybe allow a second 'follow-up'-link to point to/identify the actual PDF, like the Merge Dialog allows changes between versions. Elsevier & Sciencedirect don't do direct downloads like f.e. Springer allows. With Springer the extension works flawlessly.

(My hypthesis/impression is that with each research field the preferred platform behavior changes. People with less contact to code use the long way because of lacking capabilities to express the actual programming issue. ... and I see myself as part of them ... currently ;)