[Feature Request] Automatic batch downloading

[Joly0]

Hey, it wouls be great if the app had some way of automatically downloading all videos, including new ones, from certain models or categories or whatever (currently only intersted in models, mainly pornhub).

I know there is the batch download feature, but as far as i can see, it would nees to be started manually. It would be great if i could create a list of models and the app checks for new videos every few hours or so and if a new one is found, downloads it (i guess the app would need to keep track of already downloaded files in the case, the files are moved or renamed or so; not sure if it does that already, havent tested that), so it doesnt loose track and doesnt download the same file again.

[EchterAlsFake]

Yes that is possible and actually not that hard to implement. However I am currently doing a little rewrite of Porn Fetch on its core logic and the UI to get Android support (finally lol), so this would not be my highest priority but I'll definitely implement it in the next release.

But you would still need to like trigger a button that then starts the process for checking the videos. I can't do that automatically, because this would take much longer to implement. But searching for the new videos from specific models and automatically downloading them is definitely possible :)

[Joly0]

Ok ,thank you for your respond. Regarding the autoamtic trigger, would it be possible to have a startup parameter or so, so i could run that on a cron job and dthe process starts like that? So i wouldnt have to trigger that manually but through a cron job or something similar?
I assume this would need some proper management, so if an automatic search/download is running, then a new process wont start and try to download the same files aswell, but i think this is something that needs to be accounted for if therre was an automatic start or not, otherwise i could click the "trigger" button like 100x or so 😅

Btw, viele Grüße aus FFM

[EchterAlsFake]

Yes, I can definitely do that, utilizing a startup Parameter

Grüße Zurück xD

[EchterAlsFake]

Hey, short question. It would be much easier to implement that in the CLI rather than in the GUI. Would that be fine for you?

[Joly0]

I guess yes, would work. Though how exactly are you planing to implement it? Maybe I could then better give my opinion on this

[EchterAlsFake]

So, for the CLI it would be pretty simple. I would make json file or something similar and make an entry for each model. Each entry contains a list of Video URLs (for this model) that have already been downloaded.

You could give a list of models into the CLI, it would iterate over all videos, save the URLs and split them in downloaded / not downloaded. Each time you run the CLI with a special parameter e.g., --update-models Porn Fetch would initialize the Generator for each Model.

Because the Generators iterate over each page of the site, each generator would always start from the newest videos. Then I need to just make a call like

if url in already_downloaded_urls:
break

And that would be it. Pretty simple. With the guy this is a little bit harder to implement and I think since this is a rare use case this would better fit into the CLI :)

[Joly0]

Got it, makes sense and is quite close to what I had in mind. If I wanted, I could add a cron job that runs the CLI command regularly.
One thing that might be useful though would be a method in the GUI to add a new model to the json file. Would this be possible?

[EchterAlsFake]

Yeah sure, I can do that, you'll also be able to add and remove model URLs from the CLI with a startup parameter, so you can run that completely automatic.

[EchterAlsFake]

Okay, I am now I would say 80% finished with the implementation.

It works like this (in practice)

You provide a list of URLs (comma seprarated), and the CLI will save each model URL along with a list of pending and downloaded videos into a json file.

You can choose to update the list of URLs. By doing this Porn Fetch will iterrate over all model URLs and fetch all video URLs for each model and add new entries to the list of "pending" videos.

This is a very efficient way, because we don't download videos yet, but still completely update the database. You can then later update the database with another option and then Porn Fetch will go over each model, extract all pending URLs and then download each video. If a video is successfully downloaded, it will be marked as downloaded and removed from the pending list.

If it failed, for whatever reason, it will continue to the next video, but log the exact error with the model URL, the video URL, the index and the full traceback what happened to a file, so that it can be inspected later.

The video will then also remain in the pending list, so that you can retry it later .

You can also sort the videos by studio / model. This works by activating the option for the directory system in the settings. This way videos will be stored in a directory with the first entry of what I can find in a video's metadata. This can be the studio, the pornstar itself or some other dude. Depends on how and where the video was uploaded.

And that's basically it. I know need to improve this approach a bit and harden it. After that I can implement the startup parameters, so that you can run it as a cron job and then it would be implemented :)

[EchterAlsFake]

Version 3.6 is out. You can try the feature now if you want :)

[Joly0]

Hey, just trying it out. Its a bit more complicated then i thought. So if i understood it correctly until now, using the CLI i have to start it, add the model(s) using their urls to the database (so option 7) or have to run using the "--add-model-to-database" parameter and then i can either in the CLI download directly, or run the CLI with --batch parameter. Am i correct? There is yet no option to add new models to the database from the GUI?

When batch downloading from the CLI, after each download it throws an error:

Exception in thread Thread-24 (download):
Traceback (most recent call last):
  File "threading.py", line 1043, in _bootstrap_inner
  File "threading.py", line 994, in run
  File "Porn_Fetch_CLI.py", line 753, in download
  File "rich\progress.py", line 1650, in remove_task
KeyError: None

Also it would be really neat, if additionaly to the url in the database i could configure a name or something, so i could specify the subfolder where videos are downloaded, preferably per model. I have seen in the settings in the CLI that i can change the output path, but thats for all files.

Other than that nice feature, will have to try it a bit more, but looks good atleast on the CLI part as of now :)

[Joly0]

Ah, also before running the --batch parameter i guess i would also have to run the --update-models parameter? Or the --update-pending-urls parameter?

[EchterAlsFake]

So, if you use the arguments for running the CLI you don't have to submit the --batch parameter.

You just use them standalone like:

./CLI --add-model-to-database
./CLI --update-pending-urls (updates the list of videos that need to be fetched. Run before --update-models)
./CLI --update-models (starts downloading the actual videos)
and so on.

With the CLI Interface it's like this.

Option 2: Adds a new URL to the database. So you can submnit one, or multiple URLs of models here
After that: (Option 3): Update the pending URLs. This basically updates the list of videos that need to be fetched.
You always want to run this, before using Option 99 to start downloading the actual videos.

The thing with the progressbar seems like a bug that I haven't encountered in development. I try to reproduce and fix that.
And yes, I'll add another option to specify the name. However, this will all be in v3.7 then.

Because compiling all release assets takes a lot of time. Almost a full day for me, so I can't just make a minor change now and update the release. I hope the base is intended how you wanted it and you can use and work with it 🙏