Skip to content

Mitaka nfp 08 june 2016 oslo with perf #435

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mak-454 wants to merge 20 commits into
base: mitaka_21st_march_base
Choose a base branch
from
Open

Mitaka nfp 08 june 2016 oslo with perf #435

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
c8279e5
Merged oslo core changes in latest branch available
mak-454 Jun 8, 2016
247a0cf
Added graph flow support for events
AkashDeepSrivastava Jun 10, 2016
3169315
Added perf with latest oslo changes
AkashDeepSrivastava Jun 10, 2016
237b876
changes while starting processes
AkashDeepSrivastava Jun 11, 2016
4adfad7
need to remove after perf test
AkashDeepSrivastava Jun 11, 2016
4bff995
merged missed methods from perf branch to this merged branch
AkashDeepSrivastava Jun 11, 2016
775d59f
more changes
AkashDeepSrivastava Jun 11, 2016
4523850
tested fb, lb, fw+lb chains and ran fw+lb 1 atf test,pending graph im…
AkashDeepSrivastava Jun 11, 2016
a636f6c
Impletemented and tested graph based event triggering in orchestrator…
AkashDeepSrivastava Jun 12, 2016
c57d750
Fixed the graph logic to not send complete event information for each…
mak-454 Jun 13, 2016
0a79cb8
corrected result object returned to modules
mak-454 Jun 13, 2016
a596c59
fixed wrong function definition to class definition
mak-454 Jun 13, 2016
f0b8365
added the check when graph event complete is invoked for non graphed …
mak-454 Jun 13, 2016
d9ca0f4
cleaned up code, added logs etc
mak-454 Jun 13, 2016
ada1376
corrected the event names and logic
mak-454 Jun 13, 2016
8d5ab7c
added neutron exchange parameter, required as we are not using neutro…
mak-454 Jun 13, 2016
5393389
fixed logic to fix issues observed during atf testing
mak-454 Jun 13, 2016
199c7cc
corrected exchange problem in configurator
AkashDeepSrivastava Jun 14, 2016
3dd4f25
commented versioning for now, gives an error of sdist installation re…
mak-454 Jun 14, 2016
a748fe4
Merge branch 'mitaka_nfp_08_june_2016_oslo_with_perf' of https://gith…
mak-454 Jun 14, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
179 changes: 151 additions & 28 deletions gbpservice/neutron/services/servicechain/plugins/ncp/node_drivers/nfp_node_driver.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@
# limitations under the License.

import eventlet
eventlet.monkey_patch()

from eventlet import greenpool
from keystoneclient import exceptions as k_exceptions
from keystoneclient.v2_0 import client as keyclient
from neutron._i18n import _LE
Expand Down Expand Up @@ -230,6 +233,9 @@ class NFPNodeDriver(driver_base.NodeDriverBase):
def __init__(self):
super(NFPNodeDriver, self).__init__()
self._lbaas_plugin = None
self.thread_pool = greenpool.GreenPool(10)

@njagadish njagadish Jun 13, 2016

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make the thread pool count a constant.

self.active_threads = []
self.sc_node_count = 0

@property
def name(self):
Expand Down Expand Up @@ -376,9 +382,20 @@ def create(self, context):
self._set_node_instance_network_function_map(
context.plugin_session, context.current_node['id'],
context.instance['id'], network_function_id)
self._wait_for_network_function_operation_completion(

# Check for NF status in a separate thread
gth = self.thread_pool.spawn(
self._wait_for_network_function_operation_completion,
context, network_function_id, operation='create')

self.active_threads.append(gth)

# At last wait for the threads to complete, success/failure/timeout
if len(self.active_threads) == self.sc_node_count:
for gth in self.active_threads:
gth.wait()

@njagadish njagadish Jun 13, 2016

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The thread function is throwing an exception if a network function goes to ERROR state. In such case, here error is not percolated to GBP to detect chain create failure. GBP thinks that chain create is successful even if one node create fails.

self.active_threads = []

def update(self, context):
context._plugin_context = self._get_resource_owner_context(
context._plugin_context)
Expand Down Expand Up @@ -657,23 +674,49 @@ def _get_service_targets(self, context):
{'service_type': service_type})
raise Exception("Service Targets are not created for the Node")

service_target_info = {'provider_ports': [], 'provider_pts': [],
'consumer_ports': [], 'consumer_pts': []}
service_target_info = {
'provider_ports': [],
'provider_subnet': None,
'provider_pts': [],
'provider_pt_objs': [],
'provider_ptg': [],
'consumer_ports': [],
'consumer_subnet': None,
'consumer_pts': [],
'consumer_pt_objs': [],
'consumer_ptg': []}

for service_target in provider_service_targets:
policy_target = context.gbp_plugin.get_policy_target(
context.plugin_context, service_target.policy_target_id)
policy_target_group = context.gbp_plugin.get_policy_target_group(
context.plugin_context, policy_target['policy_target_group_id'])
port = context.core_plugin.get_port(
context.plugin_context, policy_target['port_id'])
port['ip_address'] = port['fixed_ips'][0]['ip_address']
subnet = context.core_plugin.get_subnet(
context.plugin_context, port['fixed_ips'][0]['subnet_id'])
service_target_info['provider_ports'].append(port)
service_target_info['provider_subnet'] = subnet
service_target_info['provider_pts'].append(policy_target['id'])
service_target_info['provider_pt_objs'].append(policy_target)
service_target_info['provider_ptg'].append(policy_target_group)

for service_target in consumer_service_targets:
policy_target = context.gbp_plugin.get_policy_target(
context.plugin_context, service_target.policy_target_id)
policy_target_group = context.gbp_plugin.get_policy_target_group(
context.plugin_context, policy_target['policy_target_group_id'])
port = context.core_plugin.get_port(
context.plugin_context, policy_target['port_id'])
port['ip_address'] = port['fixed_ips'][0]['ip_address']
subnet = context.core_plugin.get_subnet(
context.plugin_context, port['fixed_ips'][0]['subnet_id'])
service_target_info['consumer_ports'].append(port)
service_target_info['consumer_subnet'] = subnet
service_target_info['consumer_pts'].append(policy_target['id'])
service_target_info['consumer_pt_objs'].append(policy_target)
service_target_info['consumer_ptg'].append(policy_target_group)

return service_target_info

Expand All @@ -685,6 +728,7 @@ def _is_node_order_in_spec_supported(self, context):
for spec in current_specs:
node_list.extend(spec['nodes'])

self.sc_node_count = len(node_list)
for node_id in node_list:
node_info = context.sc_plugin.get_servicechain_node(
context.plugin_context, node_id)
Expand All @@ -707,9 +751,65 @@ def _is_node_order_in_spec_supported(self, context):
raise InvalidNodeOrderInChain(
node_order=allowed_chain_combinations)

def _get_consumers_for_provider(self, context, provider):

@njagadish njagadish Jun 13, 2016

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the advantage of getting these details here Vs in orchestrator? Is this only to avoid multiple REST calls for other data fields in "nfp_create_nf_data" variable? and to keep all the information retrieval of variable in one place?

'''
{
consuming_ptgs_details: [{'ptg': <>, 'subnets': <>}]
consuming_eps_details: []
}
'''

consuming_ptgs_details = []
consuming_eps_details = []

provided_prs_id = provider['provided_policy_rule_sets'][0]

@ashutosh-mishra ashutosh-mishra Jun 13, 2016

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we have multiple prs, and first one is allow prs ?
We need consumers only corresponding to the redirect prs.

provided_prs = context.gbp_plugin.get_policy_rule_set(
context.plugin_context, provided_prs_id)
consuming_ptg_ids = provided_prs['consuming_policy_target_groups']
consuming_ep_ids = provided_prs['consuming_external_policies']

consuming_ptgs = context.gbp_plugin.get_policy_target_groups(
context.plugin_context, filters={'id':consuming_ptg_ids})
consuming_eps_details = context.gbp_plugin.get_external_policies(
context.plugin_context, filters={'id': consuming_ep_ids})

for ptg in consuming_ptgs:
subnet_ids = ptg['subnets']
subnets = context.core_plugin.get_subnets(context.plugin_context, filters={'id':subnet_ids})
consuming_ptgs_details.append({'ptg':ptg, 'subnets':subnets})

return consuming_ptgs_details, consuming_eps_details


def _create_network_function(self, context):
"""
nfp_create_nf_data :-

{'resource_owner_context': <>,
'service_chain_instance': <>,
'service_chain_node': <>,
'service_profile': <>,
'service_config': context.current_node.get('config'),
'provider': {'pt':<>, 'ptg':<>, 'port':<>, 'subnet':<>},
'consumer': {'pt':<>, 'ptg':<>, 'port':<>, 'subnet':<>},
'management': {'pt':<>, 'ptg':<>, 'port':<>, 'subnet':<>},
'management_ptg_id': <>,
'network_function_mode': nfp_constants.GBP_MODE,
'tenant_id': <>,
'consuming_ptgs_details': [],
'consuming_eps_details': []
}

"""
nfp_create_nf_data = {}

sc_instance = context.instance
service_targets = self._get_service_targets(context)

consuming_ptgs_details, consuming_eps_details = \
self._get_consumers_for_provider(context,
service_targets['provider_ptg'][0])

if context.current_profile['service_type'] == pconst.LOADBALANCER:
config_param_values = sc_instance.get('config_param_values', {})
if config_param_values:
Expand All @@ -727,35 +827,58 @@ def _create_network_function(self, context):
context.core_plugin.update_port(
context.plugin_context, provider_port['id'], port)

port_info = []
if service_targets.get('provider_pts'):
# Device case, for Base mode ports won't be available.
port_info = [
{
'id': service_targets['provider_pts'][0],
'port_model': nfp_constants.GBP_PORT,
'port_classification': nfp_constants.PROVIDER,
}
]
if service_targets.get('consumer_ports'):
port_info.append({
'id': service_targets['consumer_pts'][0],
'port_model': nfp_constants.GBP_PORT,
'port_classification': nfp_constants.CONSUMER,
})
network_function = {
'tenant_id': context.provider['tenant_id'],
'service_chain_id': sc_instance['id'],
'service_id': context.current_node['id'],
'service_profile_id': context.current_profile['id'],
'management_ptg_id': sc_instance['management_ptg_id'],
provider = {
'pt': service_targets.get('provider_pt_objs', [None])[0],
'ptg': service_targets.get('provider_ptg', [None])[0],
'port': service_targets.get('provider_ports', [None])[0],
'subnet': service_targets.get('provider_subnet', None),
'port_model': nfp_constants.GBP_PORT,
'port_classification': nfp_constants.PROVIDER}

consumer_pt = None
consumer_ptg = None
consumer_ports = None

if service_targets['consumer_pt_objs']:

@njagadish njagadish Jun 13, 2016 •
edited
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we call these consumer_pt_instances instead of consumer_pt_objs

consumer_pt = service_targets.get('consumer_pt_objs', [None])[0]

@ashutosh-mishra ashutosh-mishra Jun 13, 2016

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

None never get executed, since we already checked with if.
Why we are doing different for both provider, consumer, Can we do in similar way

if service_targets['consumer_ptg']:
consumer_ptg = service_targets.get('consumer_ptg', [None])[0]
if service_targets['consumer_ports']:
consumer_ports = service_targets.get('consumer_ports', [None])[0]

consumer = {
'pt': consumer_pt,
'ptg': consumer_ptg,
'port': consumer_ports,
'subnet': service_targets.get('consumer_subnet', None),
'port_model': nfp_constants.GBP_PORT,
'port_classification': nfp_constants.CONSUMER}

management = {
'pt': None,
'ptg': None,
'port': None,
'subnet': None,
'port_model': nfp_constants.GBP_NETWORK,
'port_classification': nfp_constants.MANAGEMENT}

nfp_create_nf_data = {
'resource_owner_context': context._plugin_context.to_dict(),
'service_chain_instance': sc_instance,
'service_chain_node': context.current_node,
'service_profile': context.current_profile,
'service_config': context.current_node.get('config'),
'port_info': port_info,
'provider': provider,
'consumer': consumer,
'management': management,
'management_ptg_id': sc_instance['management_ptg_id'],
'network_function_mode': nfp_constants.GBP_MODE,
}
'tenant_id': context.provider['tenant_id'],
'consuming_ptgs_details': consuming_ptgs_details,
'consuming_eps_details': consuming_eps_details}

return self.nfp_notifier.create_network_function(
context.plugin_context, network_function=network_function)['id']
context.plugin_context, network_function=nfp_create_nf_data)['id']

def _set_node_instance_network_function_map(
self, session, sc_node_id, sc_instance_id, network_function_id):
Expand Down
Empty file removed gbpservice/neutron/tests/unit/nfp/config_orchestrator/agent/__init__.py
View file
Open in desktop
Empty file.
Loading