首页 / 浏览问题 / 云GIS / 问题详情
使用其他的分布式计算集群报错
2EXP 2022年10月19日

使用产品:iserver 11i 操作系统:win10 x64 问题详细描述:原来使用的本地spark,现在改为使用其他的分布式计算集群Spark,用示例数据做缓冲分析提示错误there is no available worker node in the spark cluster spark://10.1.71.1:7077。

1个回答

您好,您检查一下集群页面worker是否添加成功并且状态为ALIVE,如果没有状态为alive的worker则会出现该报错。如果目前并没有可用worker的话,您可以在主节点iserver中通过集群-》加入集群-》添加报告器并保存的方式加入集群(注意需要勾选是否分布式分析节点),本机部署服务地址使用主节点本机地址,实际使用可将地址改为实际ip和端口。spark正常状态如下:

3,148EXP 2022年10月19日

我配置的是外部集群也需要加入集群吗,这个要怎么加。

需要加入,加入方式和自带集群一致,我上面有提到:在主节点iserver中通过集群-》加入集群-》添加报告器并保存的方式加入集群(注意需要勾选是否分布式分析节点),本机部署服务地址使用主节点本机地址,实际使用可将地址改为实际ip和端口。

加入不进来呀

在集群体系中,任意一个 iServer 都可作为一个集群服务器,接受子节点的加入。子节点加入集群时,需要向集群服务器的地址报告。

集群服务器地址:http://<ip>:<port>/iserver/services/cluster

通过向集群服务器报告,可以将本机添加到集群中。加入的是配置了spark集群的iserver服务器地址,比如您上面的http://localhost:8090/iserver/services/cluster,然后把是否分布式分析节点勾选上,点击保存

您好,本地的我试过没有问题。我现在想做的是用我之前部署好的一个spark集群,上面没有安装iserver。还是说这个配置还是保留跟之前一样,不需要修改,我原来是如下图配置的。

配置集群页面选择使用其他的分布式计算集群即可,加入集群方法一致,可以不用修改。

按您说的修改完可以执行分析了,但是没有分析结果出不来。

这个不是集群配置问题哈,参考分布式分析日志具体排查失败原因。分布式分析日志路径可以自行设置,默认位置为:../logs/distributeanalysis.log
您好,日志如下,这个没有办法排查呀。

2022-10-19 14:41:04 - INFO - dataFileCache open start  
2022-10-19 17:20:46 - INFO - dataFileCache open start  
2022-10-19 17:41:31 - INFO - dataFileCache open start

感觉还是集群有问题,我已经是连外部的spark集群,但是控制台这里一直提示无法连上这个iserver自带的spark地址

您看一下执行任务时network里面对应应用ID.json,里面是否有报错消息

这种请求很多,其中个别请求会有异常,具体错误信息如下:

Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, 10.1.71.2, executor 2): java.io.StreamCorruptedException: invalid stream header: 73720025
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:354)
    at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:63)
    at org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:63)
    at org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:126)
    at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:113)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:370)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

直接查看最后一条请求,可以直接把请求的json在浏览器中打开,把整个json发给我吧,另外检查一下spark的日志,看看有没有执行该任务以及是否有出现报错。

错误信息如下,会不会是spark访问不到这个示例数据导致的呢?另外从本地spark改为外部spark之后,iServer控制台一直提示连接master异常信息(上面有截图),一段时间后iServer程序会自动退出。

{"id":"3c97fd38_9111_4f97_a93d_19ec1c8705ff","state":{"errorStackTrace":null,"endState":true,"startTime":1666249722995,"endTime":1666249815938,"publisherelapsedTime":0,"runState":"FAILED","errorMsg":"Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, 10.1.71.2, executor 2): java.io.StreamCorruptedException: invalid stream header: 73720025\n\tat java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)\n\tat java.io.ObjectInputStream.<init>(ObjectInputStream.java:354)\n\tat org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:63)\n\tat org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:63)\n\tat org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:126)\n\tat org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:113)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:370)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n\nDriver stacktrace:","elapsedTime":90401},"setting":{"DEFAULT_MASTER_ADRESS":"local[*] ","referToken":"NmTs56U0xlv0_ctrsNGgKLRtPZMDBUaO9Df9xCDAHZhaPtjoLYS_4qYUAS_2E70AzDc9W_OsehQx6e7ngGFJTg..","mainClass":null,"appName":null,"serviceInfo":null,"output":{"resultEpsgCode":3857,"outputPath":null,"datasourcePath":"E:\\soft\\supermap-iserver-11.0.0-windows-x64\\webapps\\iserver\\processingResultData\\Analyst\\udbs\\ce0671fd_a6ac_43d6_89e6_0821fe924800","datasetName":"analystResult","type":"UDB"},"args":null,"input":{"datasetName":"samples_newyork_taxi_2013-01_14k","numSlices":0,"specField":null,"datasetInfo":{"metaInfo":"{\n    \"HasHeader\": false,\n    \"FieldInfos\": [\n        {\n            \"name\": \"col0\",\n            \"type\": \"WTEXT\"\n        },\n        {\n            \"name\": \"col1\",\n            \"type\": \"WTEXT\"\n        },\n        {\n            \"name\": \"col2\",\n            \"type\": \"WTEXT\"\n        },\n        {\n            \"name\": \"col3\",\n            \"type\": \"INT32\"\n        },\n        {\n            \"name\": \"col4\",\n            \"type\": \"WTEXT\"\n        },\n        {\n            \"name\": \"col5\",\n            \"type\": \"WTEXT\"\n        },\n        {\n            \"name\": \"col6\",\n            \"type\": \"WTEXT\"\n        },\n        {\n            \"name\": \"col7\",\n            \"type\": \"INT32\"\n        },\n        {\n            \"name\": \"col8\",\n            \"type\": \"INT32\"\n        },\n        {\n            \"name\": \"col9\",\n            \"type\": \"DOUBLE\"\n        },\n        {\n            \"name\": \"X\",\n            \"type\": \"DOUBLE\"\n        },\n        {\n            \"name\": \"Y\",\n            \"type\": \"DOUBLE\"\n        },\n        {\n            \"name\": \"col12\",\n            \"type\": \"DOUBLE\"\n        },\n        {\n            \"name\": \"col13\",\n            \"type\": \"DOUBLE\"\n        }\n    ],\n    \"GeometryType\": \"POINT\",\n    \"StorageType\": \"XYColumn\",\n    \"PrjCoordsys\": 4326\n}","firstRowIsHead":false,"xIndex":10,"available":true,"name":"samples_newyork_taxi_2013-01_14k","prjCoordsys":0,"yIndex":11,"fieldInfo":[{"name":"col0","type":"WTEXT"},{"name":"col1","type":"WTEXT"},{"name":"col2","type":"WTEXT"},{"name":"col3","type":"INT32"},{"name":"col4","type":"WTEXT"},{"name":"col5","type":"WTEXT"},{"name":"col6","type":"WTEXT"},{"name":"col7","type":"INT32"},{"name":"col8","type":"INT32"},{"name":"col9","type":"DOUBLE"},{"name":"X","type":"DOUBLE"},{"name":"Y","type":"DOUBLE"},{"name":"col12","type":"DOUBLE"},{"name":"col13","type":"DOUBLE"}],"type":"CSV","separator":",","url":"../../samples/data/ProcessingData/newyork_taxi_2013-01_14k.csv"}},"sparkLogFile":null,"analyst":{"distanceUnit":"Meter","distance":"1","bounds":"","distanceField":"","dissolveField":""},"contextSetting":null,"referServicesAddress":"http://10.1.88.4:8090/iserver","referServicesType":"ISERVER"}}

您用的示例数据是个csv文件,可以看看文件是否有权限;如果同样的示例数据同样的参数操作本地spark执行成功,但是自己搭建的spark执行失败的话应该还是搭建的spark的问题。您自己搭建的Spark 的版本需要与 iServer 内置的一致,同时,Hadoop 版本也需要与其对应。iserver自带的spark版本以及Hadoop 版本可以在iserver安装包\support\spark\README.md文件中查看。例如iserver内置版本为:Spark 2.3.4 built for Hadoop 2.7.3,因此使用的 Hadoop 版本应为 2.7.3

...