CKAN_dataset plugin problem in "on_post_product_spec_validation" method

[Urtza2]

Good morning,
I am trying to do a Product Specification using as Digital Asset, CKAN Dataset.
For that I have to load CKAN-Dataset plugin. I install the latest one because it has support with IDM 7.
This plugin support Umbrella too but for me it is not necessary.
When I was creating a Product Specification I found an error or an understanding for me ;-) in ckan_dataset.py file in "on_post_product_spec_validation" method.
In this pluging there is a setting.py file where you can put variables like:
AUTH_METHOD = 'idm'
Looking into ckan_dataset.py nobody call to this method,

def _get_api_client(self, url):
        # Return API Client (Umbrella or keystone) depending on the authorization mechanism
        return self._clients[AUTH_METHOD](url)

After in on_post_product_spec_validation method it puts:

  def on_post_product_spec_validation(self, provider, asset):
        
	  
	# Read CKAN dataset resources in order to determine the broker URLs
        token = User.objects.get(username=provider.name).userprofile.access_token
        dataset_info = self._get_dataset_info(asset.get_url(), token)['result']
	print ("on_post_product_spec_validation")
        for resource in dataset_info['resources']:
            # If the CKAN resource is a URL, save it in order to enable activation and accounting
            if 'url' in resource and len(resource['url']) > 0:

                # Validate that the service exists and is secured with API Umbrella
                **app_id = self._check_dataset_api(resource['url'], provider.name)**

                if 'resources' not in asset.meta_info:
                    asset.meta_info['resources'] = []

                asset.meta_info['resources'].append({
                    'url': resource['url'],
                    'app_id': app_id
                })

                # Check that the provided role is valid for the given API service
                self._check_role(app_id, asset.meta_info['role'], resource['url'])

        asset.save()

And in check_dataset_api(resource['url'], provider.name) method in third line appear reference to umbrella client to get an app_id and the funcion return app_id
If I have not got Umbrella the code returns error.
:

def _check_dataset_api(self, url, provider_name):
      parsed_url = urlparse(url)
      server = '{}://{}'.format(parsed_url.scheme, parsed_url.netloc)

      **umbrella_client = UmbrellaClient(server)
      app_id = umbrella_client.validate_service(parsed_url.path)

      if app_id is None:
          # Raise an error since this plugin only supports FIWARE IDM role authorization
          raise PluginError('The dataset resource {} is not configured to support idm authentication'.format(url))**

      # Check if the provided app_id is valid for the included keystone instance
      keystone_client = self._get_keystone_client(url)
      keystone_client.check_ownership(app_id, provider_name)

      return app_id

If I haven't got Umbrella where I get the app id?

I thinks so that in def_check_dataset_api should be a call to def _get_api_client(self, url): and depending on that follow.

Please, I have amn European Project review in 22th October and I would like to show all of this working properly.
I have IDM KeyRock v7.01 and Biz 6.4.1 versions.
Someone can help me with this.?
Thanks
Urtza

[fdelavega]

Hi,

What kind of digital asset do you want to create?What proxy are you using for securing the services?

The CKAN-API-Dataset, is intended to monetize a CKAN dataset which includes at least one NGSI query secured with API Umbrella.

In this regard, the plugin is accessing to API Umbrella to check if the NGSI queries exist and are valid. Then it gets the client ID in know what is the IDM application which define the access role (and validate that the role actually exists).

[Urtza2]

Hi,
I have selected the digital asset from the list. It gives me Ckan dasatet api obtained after installing Ckan dataset plugin for BAE.
I am not using any proxy, I have got installed PEP proxy, how can use it?
I only want publish from CKAN and in the manual puts that it is neccesary a offering from BAE. For that I am trying to make an offering and select CKAN plugin dataset.
I didn't know that I was neccesary NGSi and Umbrella too. I know that NGSI query is necessary for Data requests. Is neccesary for make an offering in BAE too?
Where I can information of all of this?? I have to install Umbrella and NGSI CKAN plugin??
Thanks in advance
Urtza

[Urtza2]

Hi again,
Here are you an attachment of my BIZ ecosystem page. Selection of digital asset.

[fdelavega]

Hi,

If you are not going to use security the easiest way is to just remove the validation for the plugin.

For creating an offering you have two options, you can use the bae publisher extension, which creates the product and the offering from the dataset management section in CKAN.
Or you can create it manually by selecting CKAN API dataset as product type. What this extension expects is the URL of the dataset in CKAN, the media type of the dataset (the resource one) and an acquisition role which is the one granted to customers when buying.

If you remove the security you can include whatever you want as acquitision role.

[Urtza2]

Hi again,
In order to remove the validation for the plugin, what is the easiest way?
I have to comment its content these methods in ckan_dataset.py?

 def on_post_product_spec_validation(self, provider, asset):
def on_post_product_offering_validation(self, asset, product_offering):

Thanks!!
Urtza

[Urtza2]

I comment this method and It works.
Urtza

[Urtza2]

Hi again,
I open this issue because I need to publish a nsgi resource. For that I installed API-Umbrella and uncomment the validation method.
I have created and API Backend for ckan and now I'm having problems in umbrella_client.py in
def validate_service(self, path) method getting app_id.

When I make a get using PostMan to
https://localhost:9222/api-umbrella/v1/apis.json&start=0&length=100
using headers:
{'headers': {u'X-Api-Key': u'SgTZHud2Y8IHKYt5EXj3HmNF6BcweKGSNV1MhPaw', u'X-Admin-Auth-Token': u'bIakl79KUzLBs8nowOyArigtlah7DrZo4fpmv0j0'}, 'verify': False}
I retrieve this information:

{
    "draw": 0,
    "recordsTotal": 1,
    "recordsFiltered": 1,
    "data": [
        {
            "backend_host": "www.ckan.com",
            "backend_protocol": "https",
            "balance_algorithm": "least_conn",
            "created_at": "2018-10-29T07:52:07Z",
            "created_by": "584e324f-daf4-43d5-b183-16df468c7c4c",
            "deleted_at": null,
            "frontend_host": "localhost",
            "name": "ckan",
            "servers": [
                {
                    "host": "www.ckan.com",
                    "port": 443,
                    "id": "ebcdb2c7-135f-4322-bdd1-92eb366189cb"
                }
            ],
            "settings": {
                "allowed_ips": null,
                "allowed_referers": null,
                "anonymous_rate_limit_behavior": null,
                "api_key_verification_level": null,
                "api_key_verification_transition_start_at": null,
                "append_query_string": "",
                "authenticated_rate_limit_behavior": null,
                "disable_api_key": null,
                "error_data": {},
                "error_templates": {},
                "http_basic_auth": null,
                "pass_api_key_header": true,
                "pass_api_key_query_param": true,
                "rate_limit_mode": null,
                "require_https": null,
                "require_https_transition_start_at": null,
                "required_roles": [
                    "provider",
                    "api-umbrella-key-creator",
                    "api-umbrella-contact-form",
                    "issue_customer"
                ],
                "required_roles_override": null,
                "id": "448946bc-a029-4037-91a6-73faf4c00b74",
                "default_response_headers": null,
                "headers": null,
                "override_response_headers": null,
                "rate_limits": null
            },
            "sort_order": 0,
            "updated_at": "2018-10-30T10:17:08Z",
            "updated_by": "584e324f-daf4-43d5-b183-16df468c7c4c",
            "url_matches": [
                {
                    "backend_prefix": "/",
                    "frontend_prefix": "/",
                    "id": "109525e3-840c-45ae-bea8-97ae851b01ca"
                }
            ],
            "version": 2,
            "id": "11c1907e-5935-48d4-89b9-14de25948df7",
            "rewrites": null,
            "sub_settings": null,
            "frontend_prefixes": "/"
        }
    ]
}

After in the validate_service method ask for:

if 'idp_app_id' in matching_elem['settings'] and len(matching_elem['settings']['idp_app_id']):
            app_id = matching_elem['settings']['idp_app_id']

But in my response I have not idp_app_id.
I have to put into Global Request Settings or Sub-URL Request Settings or Advanced Requests Rewriting or Advanced Settings ?
If I put "app_id" hardcoded the offer is being published correctly.

Thanks a lot.
Urtza

[Urtza2]

Hi, again
How can I set an external provider app using Umbrella?

I have read about disabling some values in api-umbrella-yml file but I ca't start api-umbrella.

please someone can help me with this issue?

Thanks

[fdelavega]

Hi,

First of all, external IDP validation is not included in default API Umbrella version, we are using the one located in:

https://github.com/apinf/api-umbrella

This version allow to configure the external IDP url within api-umbrella.yml and then you will be able to include the app_id of the API settings using the Umbrella GUI