Connect externally to LoadBalanced IP.

[jwahsnakupaku]

General Question

Hi,

I've spun up NebulaGraph using the Helm chart on an on-premise k8s cluster, however I'm having some difficulty connecting to it using the Spark connector via an external spark cluster (ie not on k8s).
What I assume is happening is spark is performing the initial connect to metad, and then is getting an internal k8s address back which I'm unable to connect to externally.

The error I'm getting is;

2025-03-02 21:44:43,569 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 36.0 (TID 973) (spark-01.internal.local executor 22): com.facebook.thrift.transport.TTransportException: java.net.UnknownHostException: nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local
	at com.facebook.thrift.transport.TSocket.open(TSocket.java:206)
	at com.facebook.thrift.transport.TFramedTransport.open(TFramedTransport.java:70)
	at com.vesoft.nebula.client.meta.MetaClient.getClient(MetaClient.java:151)
	at com.vesoft.nebula.client.meta.MetaClient.freshClient(MetaClient.java:179)
	at com.vesoft.nebula.client.meta.MetaClient.getSpace(MetaClient.java:242)
	at com.vesoft.nebula.connector.nebula.MetaProvider.getVidType(MetaProvider.scala:66)
	at com.vesoft.nebula.connector.writer.NebulaWriter.<init>(NebulaWriter.scala:42)
	at com.vesoft.nebula.connector.writer.NebulaVertexWriter.<init>(NebulaVertexWriter.scala:23)
	at com.vesoft.nebula.connector.writer.NebulaVertexWriterFactory.createWriter(NebulaSourceWriter.scala:27)
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:407)
	at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:358)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.net.UnknownHostException: nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:607)
	at com.facebook.thrift.transport.TSocket.open(TSocket.java:201)
	... 18 more

2025-03-02 21:44:43,603 WARN scheduler.TaskSetManager: Lost task 0.1 in stage 36.0 (TID 974) (spark-01.internal.local executor 22): com.vesoft.nebula.client.meta.exception.ExecuteFailedException: Execute failed: Get tag execute failed, errorCode: E_TAG_NOT_FOUND
	at com.vesoft.nebula.client.meta.MetaClient.getTag(MetaClient.java:331)
	at com.vesoft.nebula.connector.nebula.MetaProvider.getTagSchema(MetaProvider.scala:101)
	at com.vesoft.nebula.connector.writer.NebulaVertexWriter.<init>(NebulaVertexWriter.scala:31)
	at com.vesoft.nebula.connector.writer.NebulaVertexWriterFactory.createWriter(NebulaSourceWriter.scala:27)
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:407)
	at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:358)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

Here's what I'm running in PySpark

nebula_config = {
    "graphd": "10.0.1.76:9669", 
    "metad": "10.0.1.76:9559",
    "user": "root",
    "password": "nebula"
}

vertices_df.write.format("com.vesoft.nebula.connector.NebulaDataSource") \
    .mode("overwrite") \
    .option("type", "vertex") \
    .option("operateType", "write") \
    .option("spaceName", "test_space") \
    .option("label", "ip") \
    .option("vidPolicy", "") \
    .option("vertexField", "_vertexId") \
    .option("batch", 5) \
    .option("metaAddress", nebula_config.get("metad")) \
    .option("graphAddress", nebula_config.get("graphd")) \
    .option("passwd", nebula_config.get("password")) \
    .option("user", nebula_config.get("user")) \
    .option("writeMode", "insert") \
    .save()

In this case 10.0.1.76 is a MetalLB loadbalanced IP I'm using, that is pointed at metad.

apiVersion: v1
kind: Service
metadata:
  name: metad-lb-service
  namespace: nebula
  labels:
    app.kubernetes.io/cluster: nebula
    app.kubernetes.io/component: graphd
    app.kubernetes.io/managed-by: nebula-operator
    app.kubernetes.io/name: nebula-graph
  annotations:
    metallb.universe.tf/allow-shared-ip: "10.0.1.76-shared"
spec:
  type: LoadBalancer
  loadBalancerIP: 10.0.1.76
  ports:
    - name: metad-thrift
      protocol: TCP
      port: 9559
      targetPort: 9559
    - name: metad-http
      protocol: TCP
      port: 19559
      targetPort: 19559
  selector:
    app.kubernetes.io/cluster: nebula
    app.kubernetes.io/component: metad
    app.kubernetes.io/managed-by: nebula-operator

Here's the values.yml I'm using with the HelmChart.

nameOverride: nebula
fullnameOverride: nebula
nebula:
  storageClassName: openebs-common-1-raid0-1
  metad:
    config:
      local_ip: 10.0.1.76
  graphd:
    config:
      local_ip: 10.0.1.76

I've checked the metad container and it has local_ip configured.

$ kubectl -nnebula  exec -it nebula-metad-1 --  grep local_ip etc/nebula-metad.conf
Defaulted container "metad" out of: metad, dynamic-flags (init)
--local_ip=10.0.1.76

However, looking at the args, looks like it might just be ignoring it as it is there as a flag?

$ kubectl -nnebula logs -f nebula-metad-0
Defaulted container "metad" out of: metad, dynamic-flags (init)
++ hostname
+ exec /usr/local/nebula/bin/nebula-metad --flagfile=/usr/local/nebula/etc/nebula-metad.conf --meta_server_addrs=nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559,nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559,nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559 --local_ip=nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local --daemonize=false

Would the args listed above override the nebula-metad.conf?

[wangajing]

I tried spark connector with nebula on k8s, and my selection is to use spark on k8s to run the job successfully. the main issue with external spark job is the network issues as you tested. for metad, as it's not stateless, it's selecting the lead node, and then return the ip to client for connecting, using loadbalancer may not work. I'm also curious if you have make it work.