Multibot autoloading from backend endpoints

[ZykaJiri]

Is your feature request related to a problem? Please describe.
I find myself switching devices or clearing browser cache for development quite frequently, which means that the local storage with all my pre-configured bots disappears and i have to add them back one by one manually.

Describe the solution you'd like
A setting which would, on dashboard load, fetch a JSON configuration with all the bot parameters from a backend endpoint. This would allow for an automated way of adding more bots and persistence between devices. An authentication method would also be necessary - bearer token or basic HTTP auth should be sufficient. Also a way to export all the currently configured bots.

Describe alternatives you've considered
A javascript injection which calls a backend endpoint and injects the corresponding bot information to localstorage.

Additional context
I am quite happy to contribute if you agree that the idea is a good fit for the ui.

[xmatthias]

Well this is technically a duplicate of #591.
It's basically asking exactly the same exact question - without adding any further detail on how this would be approached from a technical perspective to alleviate the concerns raised there.

---

So before starting to do anything -
how'd you imagine this to work from a technical perspective?
where would you persist the information?

There is no "main backend" for freqtrade/freqUI - where you can store generic information not tied to a specific bot - each UI can connect to each bot (assuming CORS is configured correctly).

So the only solution that jumps my mind at this point is to push auth information for all bots to one bot (or to all bots?)
The first would make one bot the master - which is wrong - as it's just as likely that "this chosen one" is replaced with a better working strategy than the current one - removing the shared information.

Pushing to all bots is dangerous - and not something i see as acceptable solution.
While it'll fix the "this bot went away" scenario - it's a risk in another usecase / scenario.
I can own 3 bots - and share 2 other bots with friends but have them all in one UI.
right now - these bots don't know about each other, and i think it should remain this way.

---

The most likely scenario i can see is the possibility to download a configuration file, which you can then play back.
this will however have to remove all Auth information (username can stay, but tokens must go) - adding the bots back, but requiring new authentication for each one individually.
This "require new auth" state is already in place for bots that have their refresh-token expired.

I'd see everything else as a security risk you have to compare to "leave your password file around unencrypted".
sure, some people might do that - but it's not behavior i'll even attempt to support.

[ZykaJiri]

how'd you imagine this to work from a technical perspective?

What I had in mind was to leave the back-end implementation solely to the user and create just loding logic in the UI. Probably just a statically served JSON file managed by whichever multiple-bot deployment process the user uses(in my case that is ansible), the reasoning behind this approach was that it would give users a powerful UI management tool, without the need to compromise any of the backend infrastructure, which they could tie to their own logic. The security concerns with that are significant, especially considering most users will probably not even bother to setup SSL and the bot keys would be unencrypted. It would also be quite difficult to setup.

Having given it some more thought I've come up with just as powerful solution, without such problems:
The JSON file with bot data could be instead uploaded to the users google drive, which uses encryption by default, the file could be further secured by using a folder password. This would allow instant exporting and importing of configs from any device and persistence without the need for a proper UI backend. The fiile could still be easily managed by ansible or any other deployment tool.

[xmatthias]

leaving the backend implementation to the user means it'll be effectively a dead piece of code which will be unmaintainable / unsupportable.

Every change can break someone's system - as you don't know how/what they're actually using and why it's breaking this case, but not the other.
That's clearly not an acceptable solution.

---

I'm fine with a "export" button (somewhere in settings or so) - which can export settings, (maybe plot configs, ... )- and the configured bots (without credentials!).
it's however a manual "upload / download" button - not something that the UI automatically attempts to fetch from "wherever". for this "wherever" we don't have a solution - and we'll not be attempting to query random URL's (maybe there's a config at localhost:xxxx/botconfig.json)- or even add code to attach to some random cloud store.
That'll just raise "i don't want to give google my data, please support ".
That's not part of the core UI tasks - and not something I'll burden this project with (also because it's providing next to no benefits).

Where / how that is stored is then up to the user - as it doesn't contain sensitive information (it'll require re-login once the configuration is added) - it'll not expose anything other than the IP/port combinations (but neither the password, nor access tokens).
It's however the user's responsibility to eventually export settings before a browser change - and then import once you are in the new browser.

[ZykaJiri]

I'm fine with a "export" button (somewhere in settings or so) - which can export settings, (maybe plot configs, ... )- and the configured bots (without credentials!).

Export button with no credentials is pretty much useless, by that point its much easier to just copy the entire pinia store to other devices/users. There is also no way to reasonably automate the process.

While I do see the security problems with storing an unencrypted export file, there have been quite a few exploits recently which targeted the browser profile files, export with credentials would not be much of a problem when the entire local storage is taken away instead. My point is that if any of the systems using the exported file were to be compromised, the attackers would most likely ignore a random export.json file(containing the access tokens) somewhere in the file system and instead just take the entire pinia store which contains those details in plain text, technically storing the access tokens on some random cloud service is much safer than storing them in persistent pinia store.

Two resonably secure solutions I can think of from here are:

1. A completely separate backend for the UI - Most likely using an encrypted sqlite database with credentials and ports for freqtrade bots and one/multiple master passwords that would decrypt these credentials. The new backend would call the bots' REST API and have separate endpoints for the frontend UI. This way the bots would never actually need to be exposed to the network.
2. UI modules - Allow for custom vue plugins that would add the option to write additional code for the UI without the need to alter the core functionality. This way the main code base would remain clean of the specific functionality and users would be given the choice to manage persistence(and other features) which ever way they wanted.

[xmatthias]

Neither of the 2 solutions you've proposed make sense. Implementation, maintenance and setup efforts are too high for what we're trying to solve.

Even if we were to support an "UI modules" plugin system - it'd definitely not be allowed to write authentication (most likely also not read - but go through the available api service) - so will not satisfy your needs.

I don't really see why the export / import doesn't work. That will cover 90% of the usecases.

It may not 100% cover your particular usecase, but there's very few people who try to automate their browser with ansible (or any other tool, for that matter) - so i see this as an edge-usecase.

[cmldexter]

If this is a request to have bots automatically show up on FreqUI, I actually implemented this but through my CICD pipeline. I can either add bots to FreqUI automatically and also remove them when I want to decom them, but honestly I wouldn't even add this as part of the core functionality because it goes outside the scope of the project in my opinion.

But that being said the way I pulled it off was by creating a pipeline that deploys FreqUI with a dummy strategy (which I actually called "Dashboard Strategy"). I didn't really research much on what it took to deploy FreqUI with absolutely no strategy and that was my quickest workaround) and I focused more on getting FreqUI deployed through Kubernetes and behind an authenticator amongst other things, so this is the TLDR:

I then built an inventory system for my pipeline that differentiates a "headless bot" from a "dashboard instance" and then on my bot deployment pipeline I added an option to assign the "headless bot" to a FreqUI "dashboard" instance and then as part of the pipeline I have a script that injects the info of the bot to FreqUI after is deployed, which then reboots that pod running FreqUI, but I also have my CICD integrated with a secret manager and that's key for you to securely pull this off and it kinda goes outside the scope of the developer in my opinion to also provide all of this in the form a plugin or something simple.

And if there's one additional recommendation I could give to anyone wanting to do this cleanly is to set up a Kubernetes cluster so you can leverage all the scalability it comes with it out the box because doing this on a docker environment is a rabbit hole I don't think is worth the effort.