[master] noVNC Console Integration

[ustcweizhou]

Description

This feature provides novnc console for virtual machines.

It is developed based on
(1) noVNC 1.1.0
(2) latest websockify
Some changes in cloudstack
(1) add a new servlet /novncconsole
(2) a websocket proxy based on websockify will be launched in CPVM.
(3) add vnc password authentication in websockify
(4) generate 'path' and decrypt it in websockify to connect to server/port and check vnc password.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

(1) it works with ubuntu 16.04/qemu 2.5
(2) If consoleproxy.sslEnabled is changed, please restart management server and CPVM (or cloud service in CPVM). It determines http or https will be used in novnc console
(3) If novnc.console.default is true,novnc console will be default console on UI. If it is false, we still can access novnc console via /novncconsole?cmd=access&vm=XXX
(4) global setting novnc.encryption.key is hidden, it can be changed via cloudmonkey. Restart CPVM (or cloud service in CPVM) if it is changed.

TODOs

• Test on KVM (CentOS + Ubuntu)
• Test on VMware
• Test on XenServer and XCP-NG
• Fix license declarations etc
• Fix end-to-end TLS/security on Chrome + Firefox

[ustcweizhou]

For someone still use cloudstack 4.11.2 in production, #3283

[DennisKonrad]

@ustcweizhou To migrate from an existing installation, do we need to build a new systemvm template? Or are the changes to the CPVM applied online somehow?

[ustcweizhou]

@DennisKonrad
python-numpy is required in CPVM. It would be nice to have it pre-installed in systemvm template. However, if CPVM has internet connection it is not a problem, because python-numpy will be installed when noVNC is launched in CPVM.

[ustcweizhou]

@rhtyd @svenvogel
This PR has not conflict with master branch, could you test it when you have time ?
especially on xenserver/vmware

[DennisKonrad]

@ustcweizhou @svenvogel We already built the code and that worked fine. We are planning to try the feature next week.

Then we can give feedback and test.

[yadvr]

Thanks @ustcweizhou I'm busy with work for at least two more weeks but I'll certainly review and help test the PR (and possible fix issues wrt xenserver/vmware testing).

[yadvr]

@ustcweizhou can you check and fix the pylint issues: (refer https://travis-ci.org/apache/cloudstack/jobs/524837786)

+'[' 1 -gt 0 ']'
+echo 'Pylint failed, please check your code'
Pylint failed, please check your code
+exit 1

It's probably best if we can fix the build to ignore novnc related source file and websockify python codebase
@blueorangutan package

[blueorangutan]

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2731

[yadvr]

@blueorangutan test

[blueorangutan]

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

Trillian test result (tid-3548)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 14744 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3296-t3548-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_public_ip_range.py
Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_routers.py
Intermittent failure detected: /marvin/tests/smoke/test_secondary_storage.py
Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 50 look OK, 20 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestResetVmOnReboot>:setup	Error	0.00	test_reset_vm_on_reboot.py
ContextSuite context=TestRAMCPUResourceAccounting>:setup	Error	0.00	test_resource_accounting.py
ContextSuite context=TestRouterDHCPHosts>:setup	Error	0.00	test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPOpts>:setup	Error	0.00	test_router_dhcphosts.py
ContextSuite context=TestISOUsage>:setup	Error	0.00	test_usage.py
ContextSuite context=TestLBRuleUsage>:setup	Error	0.00	test_usage.py
ContextSuite context=TestNatRuleUsage>:setup	Error	0.00	test_usage.py
ContextSuite context=TestPublicIPUsage>:setup	Error	0.00	test_usage.py
ContextSuite context=TestSnapshotUsage>:setup	Error	0.00	test_usage.py
ContextSuite context=TestVmUsage>:setup	Error	0.00	test_usage.py
ContextSuite context=TestVolumeUsage>:setup	Error	0.00	test_usage.py
ContextSuite context=TestVpnUsage>:setup	Error	0.00	test_usage.py
ContextSuite context=TestRouterDns>:setup	Error	0.00	test_router_dns.py
ContextSuite context=TestRouterDnsService>:setup	Error	0.00	test_router_dnsservice.py
ContextSuite context=TestRouterIpTablesPolicies>:setup	Error	0.00	test_routers_iptables_default_policy.py
ContextSuite context=TestVPCIpTablesPolicies>:setup	Error	0.00	test_routers_iptables_default_policy.py
test_01_isolate_network_FW_PF_default_routes_egress_true	Error	0.10	test_routers_network_ops.py
test_02_isolate_network_FW_PF_default_routes_egress_false	Error	0.10	test_routers_network_ops.py
ContextSuite context=TestRedundantIsolateNetworks>:setup	Error	1.22	test_routers_network_ops.py
ContextSuite context=TestRouterServices>:setup	Error	0.00	test_routers.py
test_01_sys_vm_start	Failure	0.07	test_secondary_storage.py
test_02_sys_template_ready	Failure	0.06	test_secondary_storage.py
ContextSuite context=TestCpuCapServiceOfferings>:setup	Error	0.00	test_service_offerings.py
ContextSuite context=TestServiceOfferings>:setup	Error	0.12	test_service_offerings.py
ContextSuite context=TestSnapshotRootDisk>:setup	Error	0.00	test_snapshots.py
test_01_list_sec_storage_vm	Failure	0.02	test_ssvm.py
test_02_list_cpvm_vm	Failure	0.02	test_ssvm.py
test_03_ssvm_internals	Failure	0.02	test_ssvm.py
test_04_cpvm_internals	Failure	0.02	test_ssvm.py
test_05_stop_ssvm	Failure	0.02	test_ssvm.py
test_06_stop_cpvm	Failure	0.02	test_ssvm.py
test_07_reboot_ssvm	Failure	0.02	test_ssvm.py
test_08_reboot_cpvm	Failure	0.02	test_ssvm.py
test_09_destroy_ssvm	Failure	0.02	test_ssvm.py
test_10_destroy_cpvm	Failure	0.02	test_ssvm.py
test_02_create_template_with_checksum_sha1	Error	65.31	test_templates.py
test_03_create_template_with_checksum_sha256	Error	65.31	test_templates.py
test_04_create_template_with_checksum_md5	Error	65.30	test_templates.py
test_05_create_template_with_no_checksum	Error	65.31	test_templates.py
test_02_deploy_vm_from_direct_download_template	Error	1.19	test_templates.py
test_03_deploy_vm_wrong_checksum	Error	1.34	test_templates.py
ContextSuite context=TestTemplates>:setup	Error	14.76	test_templates.py
ContextSuite context=Test01DeployVM>:setup	Error	0.00	test_vm_life_cycle.py
ContextSuite context=Test02VMLifeCycle>:setup	Error	0.00	test_vm_life_cycle.py
ContextSuite context=Test03SecuredVmMigration>:setup	Error	0.00	test_vm_life_cycle.py
ContextSuite context=TestVmSnapshot>:setup	Error	0.04	test_vm_snapshots.py
ContextSuite context=TestCreateVolume>:setup	Error	0.00	test_volumes.py
ContextSuite context=TestVolumes>:setup	Error	0.00	test_volumes.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	Error	4.67	test_vpc_redundant.py
test_02_redundant_VPC_default_routes	Error	3.63	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	Error	3.64	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	Error	2.62	test_vpc_redundant.py
test_05_rvpc_multi_tiers	Error	3.65	test_vpc_redundant.py
test_01_VPC_nics_after_destroy	Error	3.63	test_vpc_router_nics.py
test_02_VPC_default_routes	Error	2.61	test_vpc_router_nics.py
test_01_redundant_vpc_site2site_vpn	Failure	1.22	test_vpc_vpn.py
test_01_vpc_site2site_vpn_multiple_options	Failure	4.25	test_vpc_vpn.py
test_01_vpc_remote_access_vpn	Failure	1.12	test_vpc_vpn.py
test_01_vpc_site2site_vpn	Failure	2.21	test_vpc_vpn.py

[DennisKonrad]

@ustcweizhou @svenvogel I did a small test with this last week but couldn't get it to work. I'm not sure but just deploying a new management server with this PR didn't work. Is there anything I needed to do to make it work?

[ustcweizhou]

@DennisKonrad have you uploaded ssl certificate for console , and made some changes in global setting for domain suffix ?

[DennisKonrad]

@ustcweizhou We do have:
consoleproxy.sslEnabled set to true
consoleproxy.url.domain is also set

I somewhere uploaded a certificate but can't remember right now where one has to do that. But yes, we uploaded a certificate.

If you mean internatl domain suffix: that's unchanged (still 'cloud.internal')
Can you provide me some more information on what steps to take to get this working?

[ustcweizhou]

@DennisKonrad
see steps as below
(1) set consoleproxy.url.domain to *.domain.suffix (or domain.suffix, depends on your dns settings), and consoleproxy.sslEnabled to true, then restart management server
(2) upload ssl certificate, cpvm and ssvm will be restarted
(3) reboot cpvm if you change global settings again

by the way, what hypervisor do you use ?

[DennisKonrad]

@ustcweizhou So the steps you provided. We have the setup like you described it already in place.

Like i wrote:
(1) consoleproxy.url.domain is already set
(2) our certificate is already uploaded
(3) no need to reboot

How can we use the novnc console without settings up things we already have in place?
We use KVM

[ustcweizhou]

@DennisKonrad
do you get error on console ? if so, could you please attach a screenshot ?

or do you still see the native console ? novnc.console.default should be true in global setting.

[DennisKonrad]

@ustcweizhou No error, just not loading anything. I will try this again tomorror and report back.

I'm not sure anymore if I correctly set the global variable. But the variable is present. If you have any questions I should check on please feel free to ask.

[yadvr]

@ustcweizhou can you fix build failure, see https://travis-ci.org/apache/cloudstack/jobs/524837786
Likely the new noVNC assets need to be removed from license and lint related checks.

[DennisKonrad]

So the error seems to be "Invalid client ip address" after looking in the novnc.log... Is this some security feature? @ustcweizhou

root@v-1-VM:~# tail -f /var/log/novnc.log 
WebSocket server settings:
  - Listen on :8080
  - Web server. Web root: /root/noVNC
  - No SSL/TLS support (no cert file)
  - Backgrounding (daemon)
  - proxying from :8080 to None:None
172.20.0.1 - - [19/Feb/2020 10:41:36] 172.20.0.1: Plain non-SSL (ws://) WebSocket connection
172.20.0.1 - - [19/Feb/2020 10:41:36] 172.20.0.1: Path: '/accc60e6[...]2f9bd836fc5'
172.20.0.1: Not accepted: Invalid client ip address

[yadvr]

@ustcweizhou do you have any plans on this? Our @davidjumani is going to work on it towards 4.15 milestone.

[weizhouapache]

So the error seems to be "Invalid client ip address" after looking in the novnc.log... Is this some security feature? @ustcweizhou

root@v-1-VM:~# tail -f /var/log/novnc.log 
WebSocket server settings:
  - Listen on :8080
  - Web server. Web root: /root/noVNC
  - No SSL/TLS support (no cert file)
  - Backgrounding (daemon)
  - proxying from :8080 to None:None
172.20.0.1 - - [19/Feb/2020 10:41:36] 172.20.0.1: Plain non-SSL (ws://) WebSocket connection
172.20.0.1 - - [19/Feb/2020 10:41:36] 172.20.0.1: Path: '/accc60e6[...]2f9bd836fc5'
172.20.0.1: Not accepted: Invalid client ip address

yes @DennisKonrad
in novnc it checks if the source ip to novnc server and management server are the same.

        # Check client ip address
        if self.client_ip != self.client_address[0]:
            self.send_close()
            raise self.server.EClose('Not accepted: Invalid client ip address')

(self.client_ip is source ip to management server, self.client_address[0] is source ip to novnc)

If you use reverse proxy or load balancer (nginx, haproxy, pfsense,etc), it might be an issue.

[weizhouapache]

@ustcweizhou do you have any plans on this? Our @davidjumani is going to work on it towards 4.15 milestone.

@rhtyd We will include current source code in our next version. it works fine for many years on our productions.

[DennisKonrad]

@rhtyd Could you elaborate on what needs to be done by @davidjumani?

[DennisKonrad]

@ustcweizhou I'll check why those ip's differ in my testing environment.
When I disable the check it works just fine

[svenvogel]

@rhtyd We will include current source code in our next version. it works fine for many years on our productions.

@weizhouapache this sounds good. 👍we support this! this is a nice feature. is this console a multilanguage?

[yadvr]

@weizhouapache that's good to know.

@DennisKonrad this PR is far from complete for general consumption, and possibly requires the following (and more): /cc @PaulAngus @davidjumani

• Test and support XenServer, VMware in addition to KVM
• Support TLS/secured vnc connection when libvirtd is secured by CA-framework for KVM
• Validate the token exchange and security concerns (I've a bunch of test cases, plan in mind)
• Add support for active session management (allow only one view per VM), the active sessions reported by CPVM agent is used by the management server to add/remove more CPVMs in a zone
• I personally prefer a java-based websockets proxy http/https server which can work with ^^ and give us more control over connection validation and security (esp with the CA framework), we probably want to reduce additional packages/dependencies. I'm also not sure if/how could python/websockify provide a way to track active sessions and manage connection security.
• Support for multiple keyboards (like the current console proxy, for example jp, uk, us, fr etc)
• While Hyper-V largely does not working out of the box, but legacy console proxy proxies RDP (I think) on Hyper-V and therefore we also need to find suitable replacement for rdp-vnc cross-over, or if newer HyperV can support native VNC (needs checking @PaulAngus)
• Provide an upgrade path for users and discuss whether in the next release we want to deprecate and remove the current console proxy or simply remove in the next (unless all existing requirements of legacy console proxy are replaced and novnc is tested in various environments, it's not best to simply remove the legacy console proxy)

[DennisKonrad]

@rhtyd I think novnc should be added as option and not as a complete replacement for the current console. The only change for that would be to not set novnc as the default in the global settings.

If you have a complete replacement of the current console in mind that should be done by additional PRs and not by moving this PR into the far future. I responded to the complete list but to see it you have to click-open.

• Test and support XenServer, VMware in addition to KVM

I'll test KVM extensively. Also I think the community will pick up this PR for the other hypervisors. Never the less I know a lot of features that are merged for usage with single hypervisor with remark to the possibilty to expand them later (just take a look at vm ingestion merged by you and boris).

• Support TLS/secured vnc connection when libvirtd is secured by CA-framework for KVM

This one I would like you to explain in more detail. If one uses CA framework you want it automatically to activate TLS for novnc? How would this work for a more complex setup like load balancing where TLS gets terminated at the load balancer?

• Validate the token exchange and security concerns (I've a bunch of test cases, plan in mind)

That's great. We also have in mind to check this now that we figured out why the console was not working in our test env.

• Add support for active session management (allow only one view per VM), the active sessions reported by CPVM agent is used by the management server to add/remove more CPVMs in a zone

In my opinion that is a feature to add later. Having multiple views per VM seems to be desirable.

• I personally prefer a java-based websockets proxy http/https server which can work with ^^ and give us more control over connection validation and security (esp with the CA framework), we probably want to reduce additional packages/dependencies. I'm also not sure if/how could python/websockify provide a way to track active sessions and manage connection security.

It more or less sounds like you do want to reimplement this PR in java so it works with the CA framework? I understand the concers for security but let's do not mix that with the integration in the CA framework.

• Support for multiple keyboards (like the current console proxy, for example jp, uk, us, fr etc)

That would indeed be great but in the "legacy" console this isn't working right now as well.

• While Hyper-V largely does not working out of the box, but legacy console proxy proxies RDP (I think) on Hyper-V and therefore we also need to find suitable replacement for rdp-vnc cross-over, or if newer HyperV can support native VNC (needs checking @PaulAngus)

This PR implements novnc as option and I think it has to be off as default as well. It's cool that you want to implement HyperV support but I don't get why the milestone of this PR moves because of that?

• Provide an upgrade path for users and discuss whether in the next release we want to deprecate and remove the current console proxy or simply remove in the next (unless all existing requirements of legacy console proxy are replaced and novnc is tested in various environments, it's not best to simply remove the legacy console proxy)

Like written above this PR is not replacing the "legacy" console. Why do you suggest deprecating the "legacy" console?

[DennisKonrad]

@DennisKonrad this PR is far from complete for general consumption, and possibly requires the following (and more): /cc @PaulAngus @davidjumani

This PR at no time seems like a drop-in replacement to me. But I'm sure it cannot even become one if we block it.

[weizhouapache]

@weizhouapache that's good to know.

@DennisKonrad this PR is far from complete for general consumption, and possibly requires the following (and more): /cc @PaulAngus @davidjumani

• Test and support XenServer, VMware in addition to KVM
• Support TLS/secured vnc connection when libvirtd is secured by CA-framework for KVM
• Validate the token exchange and security concerns (I've a bunch of test cases, plan in mind)
• Add support for active session management (allow only one view per VM), the active sessions reported by CPVM agent is used by the management server to add/remove more CPVMs in a zone
• I personally prefer a java-based websockets proxy http/https server which can work with ^^ and give us more control over connection validation and security (esp with the CA framework), we probably want to reduce additional packages/dependencies. I'm also not sure if/how could python/websockify provide a way to track active sessions and manage connection security.
• Support for multiple keyboards (like the current console proxy, for example jp, uk, us, fr etc)
• While Hyper-V largely does not working out of the box, but legacy console proxy proxies RDP (I think) on Hyper-V and therefore we also need to find suitable replacement for rdp-vnc cross-over, or if newer HyperV can support native VNC (needs checking @PaulAngus)
• Provide an upgrade path for users and discuss whether in the next release we want to deprecate and remove the current console proxy or simply remove in the next (unless all existing requirements of legacy console proxy are replaced and novnc is tested in various environments, it's not best to simply remove the legacy console proxy)

@rhtyd good you have many nice ideas !
Maybe we can create a SIG for it.

[weizhouapache]

@rhtyd We will include current source code in our next version. it works fine for many years on our productions.

@weizhouapache this sounds good. 👍we support this! this is a nice feature. is this console a multilanguage?

@svenvogel we had some research to support multiple language keyboard in the past.
it is based on a open source project https://github.com/ustcweizhou/mok-project
however it is not deployed to our production.

[weizhouapache]

@DennisKonrad this PR is far from complete for general consumption, and possibly requires the following (and more): /cc @PaulAngus @davidjumani

This PR at no time seems like a drop-in replacement to me. But I'm sure it cannot even become one if we block it.

@rhtyd @DennisKonrad @DaanHoogland @PaulAngus @svenvogel
just a proposal, can we add some features or improvements as "Beta features" or "Alpha features"?

[yadvr]

@ustcweizhou a SIG may not be necessary as this would most likely require few weeks (not months), once @davidjumani start this sometime next week any collaboration would be welcomed on the public PR. The bare minimum requirement would be that it works on all hypervisors with SSL/TLS certificates, in addition to the legacy console proxy.

[DaanHoogland]

@rhtyd @DennisKonrad @DaanHoogland @PaulAngus @svenvogel
just a proposal, can we add some features or improvements as "Beta features" or "Alpha features"?

@weizhouapache I am all for it 👍. We don't do that enough. However we must think of a way to clearly mark features like than, and not be afraid to retract them if they turn out not ok. The, or an important thing to note there is that retracting is a backward incompatibility that we take on.

[weizhouapache]

a SIG may not be necessary as this would most likely require few weeks (not months), once @davidjumani start this sometime next week any collaboration would be welcomed on the public PR. The bare minimum requirement would be that it works on all hypervisors with SSL/TLS certificates, in addition to the legacy console proxy.

@rhtyd good. expect to see it in coming few weeks.

[yadvr]

@ustcweizhou just to update @davidjumani has already started work on it and I'll be supporting him, we'll most likely start a draft PR in a week's time.

[weizhouapache]

@ustcweizhou just to update @davidjumani has already started work on it and I'll be supporting him, we'll most likely start a draft PR in a week's time.

@rhtyd good, thanks.

[DaanHoogland]

@weizhouapache I'm closing this in favour of #3296. please re-open if I'm at fault.