merge dev new features to es 7.4 dev

README.MD

@@ -1,4 +1,4 @@
-From Graph to Elastic - A tail of a Yang-DB...
+From Graph to Open(Elastic)Search - A tail of a Yang-DB...
 =========================================================================================
 
 [![Coverage Status](https://coveralls.io/repos/github/YANG-DB/yang-db/badge.svg?branch=develop)](https://coveralls.io/github/YANG-DB/yang-db?branch=develop)
@@ -14,6 +14,11 @@ From Graph to Elastic - A tail of a Yang-DB...
 * Download [yang.db] (https://github.com/YANG-DB/yang-db) latest docker release docker pull yangdb/yang.db
 * Run: docker run -p 8888:88 --net=host -it yangdb/yang.db:latest (https://hub.docker.com/r/yangdb/yang.db/tags)
 
+#### Latest News
+ * opensearch community lecture given on yangDb
+ * https://github.com/YANG-DB/yang-db/blob/develop/docs/info/YangDbOpenSearch.pdf
+ * https://github.com/YANG-DB/yang-db/blob/develop/docs/info/KnowledgeGraphDeck.md
+
 
 #### Project YANG-DB
 * http://www.yangdb.org/ 
@@ -56,6 +61,7 @@ Infrastructure Technologies
 -----------------------------------------------------------------------------------------------
 
 * [elasticsearch](https://www.elastic.co/)
+* [opensearch](https://opensearch.org/)
 * [jooby](https://jooby.org/)
 * [tinkerpop](http://tinkerpop.apache.org/)
 
@@ -127,7 +133,7 @@ One major difference in this model (compared to the strict relational schema) is
 Additional graph model is the Resource Description Framework (RDF) model.
 
 
-Why Elastic
+Why Elastic / Opensearch
 ===========
 ![](https://media.licdn.com/dms/image/C4E12AQFR7RVJvIIVxA/article-inline_image-shrink_1000_1488/0?e=1547683200&v=beta&t=thjzxR2XgRNw1ldZCaRXzn4K7Nuc-giG0zAdtUIW7XU)
 
@@ -141,12 +147,12 @@ social graph that must be frequently updated and available for both:
 All the read & write are made in concurrency with reasonable response
 time and ever growing throughput.
 
-The first requirement was fulfilled using Elasticsearch – a well
+The first requirement was fulfilled using the DB  – a well
 known and established NoSql document search and storage engine capable
 of containing very large volume of data.
 
 For the second requirement **we decided that our best solution would be to
-use elasticsearch as the non-native graph-DB storage layer**.
+use opensearch as the non-native graph-DB storage layer**.
 
 As mentioned before, a graph-DB storage layer can be implemented using a
 non-native storage such as NoSql storage.
@@ -161,11 +167,11 @@ Modeling data as graph
 
 The first issue on our plate is to design the data model representing the graph, as a set of vertices and edges.
 
-With elastic we can utilize its powerful search abilities to efficiently fetch node & relation documents according to the query filters.
+With opensearch we can utilize its powerful search abilities to efficiently fetch node & relation documents according to the query filters.
 
-##### Elastic Index
+##### opensearch Index
 
-In elasticsearch each index can be described as a table for a specific
+In opensearch each index can be described as a table for a specific
 schema, the index itself is partitioned into shared which allow scale
 and redundancy (with replicas) across the cluster.
 
@@ -174,9 +180,9 @@ following formula:
 
 *shard_num = hash(_routing) % num_primary_shards*
 
-Each index has a schema (called type in elastic) which defines the
-documents structure (called mapping in elastic). Each index can hold
-only a single type of mapping (since elastic 6)
+Each index has a schema (called type in opensearch) which defines the
+documents structure (called mapping in opensearch). Each index can hold
+only a single type of mapping 
 
 The vertices index will contain the vertices documents with the
 properties, the edges index will contain the edges documents with their
@@ -225,9 +231,9 @@ From logical to physical
 ------------------------
 
 Once a logical query is given we need to translate it to the physical layer
-of the data storage which is elasticsearch.
+of the data storage which is opensearch.
 
-Elastic has a query DSL which is focused on search and aggregations –
+opensearch has a query DSL which is focused on search and aggregations –
 not on traversing, we need an additional translation phase that will
 take into account the schematic structure of the graph (and the
 underlying indices).
@@ -239,7 +245,7 @@ steps:
 
 -   translating the labels into real schema entities (indices)
 
--   creating the physical elastic query
+-   creating the physical opensearch query
 
 This is the process in a high-level review, in practice - there will be
 more stages that optimize the logical query; in some cases it is
@@ -252,9 +258,9 @@ Conclusion
 ---------------
 We started with discussing the purpose of graphs DB in today’s business use cases and reviewed different models for representing a graph. Understanding the fundamental logical building blocks that a potential graph DB should consist and discussed an existing NoSql candidate to fulfill the storage layer requirements.
 
-Once we selected elasticsearch as the storage layer we took the LDBC Social Network Benchmark graph model and simplified it to be optimized in that specific storage. We discussed the actual storage schema with the redundant properties and reviewed cypher language to query the storage in an sql-like graph pattern language.
+Once we selected opensearch as the storage layer we took the LDBC Social Network Benchmark graph model and simplified it to be optimized in that specific storage. We discussed the actual storage schema with the redundant properties and reviewed cypher language to query the storage in an sql-like graph pattern language.
 
-We continued to see the actual transformation of the cypher query into a physical execution query that will run by Elasticsearch.
+We continued to see the actual transformation of the cypher query into a physical execution query that will run by opensearch.
 
 In the last section we took a simple graph query and drilled down into the details of the execution strategies and the bulking mechanism.
 
@@ -265,3 +271,27 @@ Start Using
  - See http://www.yangdb.org/general-info.html
  - See http://www.yangdb.org/docker.html
  - See http://www.yangdb.org/get-involved.html
+
+Please review the following tutorial pages:
+---------------
+Installaiton tutorial:
+
+ - https://github.com/YANG-DB/yang-db/blob/develop/docs/tutorial/sample/dragons/installation.md
+
+Schema creation tutorial:
+
+ - https://github.com/YANG-DB/yang-db/blob/develop/docs/tutorial/sample/dragons/create-ontology.md
+
+Data Loading tutorial:
+
+ - https://github.com/YANG-DB/yang-db/blob/develop/docs/tutorial/sample/dragons/load-data.md
+
+Query the Graph tutorial:
+
+ - https://github.com/YANG-DB/yang-db/blob/develop/docs/tutorial/sample/dragons/query-the-data.md
+
+Projection materialization & count tutorial:
+
+ - https://github.com/YANG-DB/yang-db/blob/develop/docs/tutorial/sample/dragons/projection-and-count.md
+
+

src/main/config/grafana/runGrafana.sh → dashboards/grafana/runGrafana.sh

@@ -1,3 +1,4 @@
+#!/bin/bash
 #docker pull grafana
 docker pull grafana/grafana
 # docker Run grafana

dashboards/kibana/runKibana.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+docker run \
+  --name kibana \
+  --publish 5601:5601 \
+  --network host \
+  --env "ELASTICSEARCH_HOSTS=http://127.0.0.1:9200" \
+  docker.elastic.co/kibana/kibana:6.5.4

docker-compose-embedded.yml → docker/docker-compose-embedded.yml

@@ -1,24 +1,13 @@
 version: '3.3'
 
 services:
-  fuse:
-    image: yangdb/yang.db:latest
-    ports:
-      - 8888:8888
-    environment:
-      - "elasticsearchHosts=localhost"
-      - "elasticsearchClusterName=Knowledge"
-    volumes:
-      - fuse-data:/usr/share/fuse
-    networks:
-      - network
-  es:
+  elasticsearch:
     image: docker.elastic.co/elasticsearch/elasticsearch:6.5.4
     environment:
       - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
       - "discovery.type=single-node"
-      - "network.publish_host=127.0.0.1"
-      - "cluster.name=${ELASTICSEARCH_CLUSTER_NAME}"
+      - "network.publish_host=http://elasticsearch:9200"
+      - "cluster.name=Dragons"
     networks:
       - network
     ports:
@@ -30,31 +19,36 @@ services:
         published: 9300
         protocol: tcp
         mode: host
-    deploy:
-      endpoint_mode: dnsrr
-      mode: 'global'
-  elasticsearch_exporter:
-    image: justwatch/elasticsearch_exporter:1.1.0
-    command:
-      - '--es.uri=http://fuse:9200'
-    restart: always
-    ports:
-      - "127.0.0.1:9114:9114"
-  grafana:
-    image: grafana/grafana
-    container_name: grafana
     volumes:
-      - grafana-data:/usr/share/grafana
+      - esdata1:/usr/share/elasticsearch/data
+  kibana:
+    image: docker.elastic.co/kibana/kibana:6.5.4
+    links:
+      - elasticsearch
+    depends_on:
+      - elasticsearch
     ports:
-      - 3000:3000
+      - 5601:5601
+    networks:
+      - network
+  fuse:
+    image: yangdb/yang.db:latest
+    depends_on:
+      - elasticsearch
+    ports:
+      - 8888:8888
+    environment:
+      - "elasticsearchHosts=localhost"
+      - "elasticsearchClusterName=${ELASTICSEARCH_CLUSTER_NAME}"
+    volumes:
+      - fuse-data:/usr/share/fuse
     networks:
       - network
-    user: "0"
 
 volumes:
   fuse-data:
     driver: local
-  grafana-data:
+  esdata1:
     driver: local
 
 networks:

docker-compose.yml → docker/docker-compose.yml

@@ -6,7 +6,7 @@ services:
     ports:
       - 8888:8888
     environment:
-      - "elasticsearchHosts=10.6.91.1"
+      - "elasticsearchHosts=127.0.0.1"
       - "elasticsearchClusterName=${ELASTICSEARCH_CLUSTER_NAME}"
     networks:
       - network

docker/runLocalDevDocker.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+clusterName=$1
+
+echo "Starting elasticsearch clusterName:$clusterName"
+
+docker network create elastic
+docker pull docker.elastic.co/elasticsearch/elasticsearch:6.5.4
+docker run -d --name elasticsearch --net elastic -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "cluster.name=$clusterName" -e "ES_JAVA_OPTS=-Xms13g -Xmx13g" docker.elastic.co/elasticsearch/elasticsearch:6.5.4
+echo " elasticsearch started"
+
+echo " Starting Kibana"
+docker pull docker.elastic.co/kibana/kibana:7.12.1
+docker run -d --name kibana --net elastic -p 5601:5601 -e "ELASTICSEARCH_HOSTS=http://elasticsearch:9200" docker.elastic.co/kibana/kibana:6.5.4
+echo " Kibana started"
+
+echo " Starting Yang-DB"
+echo "docker run --network host  -it  yangdb/yang.db:${clusterName}"
+
+#run docker locally with port exposed - option 1
+docker run  -d --net elastic  -it  yangdb/yang.db:${clusterName}
+echo " Yang-DB started"

docs/branch/analytics_support.md

@@ -0,0 +1,43 @@
+Abstract: 
+
+The purpose of this branch is to integrate the E/S transformer API into yandDb offering advance analytics 
+
+**Details**
+
+The transform API defines a transform, which copies data from source indices, transforms it, and persists it into an entity-centric destination index. The entities are defined by the set of group_by fields in the pivot object.
+
+You can also think of the destination index as a two-dimensional tabular data structure (known as a data frame).
+
+Every YangDb Entity/Relation has a logical representation in an ontology structure, We would like to allow the ability to group and collect common features from an entity/relation into a materialized index that will optimize certain aggregation query and will reflect some informative feature that is otherwise available only using query.
+
+Lets review the next sample...
+
+Node type Person with the following fields: 
+
+ * id
+ * name
+ * gender
+ * birth date
+ * birth location
+ * family status
+
+We also have the following relation of type friend (person to person):
+
+ * source Id
+ * source Name
+ * target Id
+ * target Name
+ * date Of Friendship
+ * friendship description
+
+Lets assume that we have an analytics that would like to express the following characteristics:
+
+* amount of people having more than 100 friends gained recently (last month)
+* amount of people born near-by (some radius) and are friends
+* amount of single people (male/female) having fewer than 10 friends
+
+These questions can be queried on demand but will drain memory and cause large resource spikes
+
+Lets assume that we would like to query these questions often and would like to materialize them for performance considerations 
+
+We can also take advantage of these materialized views in order to more efficiently join this information directly with standard yangDb queries... 

docs/branch/es7_4_2-support.md

@@ -0,0 +1,23 @@
+Abstract: 
+
+The purpose of this branch is to integrate the E/S version 7.4.2 into yandDb 
+
+Changes: 
+
+    Upgrade jooby version to 1.6.8
+
+    Upgrade Elasticsearch version to 7.4.2
+
+Fixes:
+
+    Remove BasicIdGenerator.getNext() needs no version lock
+    Fix CSV-convertor to concider LOV (dictionary) when creating the json document
+
+    Fix ES: 
+        - remove "_all" mapping    
+        - remove standard filter
+        - fix ngram max-min diff
+
+Upgrades: 
+
+    Add support for retry using https://reflectoring.io/retry-with-resilience4j/