2014-06-12 00:33:24,608 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: SKEWED_JOIN
2014-06-12 00:33:24,609 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, DuplicateForEachColumnRewrite, GroupByConstParallelSetter, ImplicitSplitInserter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]}
2014-06-12 00:33:24,616 [main] INFO org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher - !!!!!!!!!! Launching Spark (woot) !!!!!!!!!!!!
2014-06-12 00:33:24,625 [main] INFO org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code.
2014-06-12 00:33:24,625 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-06-12 00:33:24,633 [main] INFO org.apache.pig.impl.builtin.PartitionSkewedKeys - input file: hdfs://localhost:54310/tmp/temp-1456742965/tmp-1456742965
2014-06-12 00:33:24,636 [main] INFO org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher - Converting operator POLoad (Name: A: Load(hdfs://localhost:54310/input/data1:PigStorage(',')) - scope-78 Operator Key: scope-78)
2014-06-12 00:33:24,650 [main] INFO org.apache.spark.storage.MemoryStore - ensureFreeSpace(63445) called with curMem=253780, maxMem=559310438
2014-06-12 00:33:24,650 [main] INFO org.apache.spark.storage.MemoryStore - Block broadcast_4 stored as values to memory (estimated size 62.0 KB, free 533.1 MB)
2014-06-12 00:33:24,652 [main] INFO org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher - Converting operator POForEach (Name: A: New For Each(false,false,false)[bag] - scope-85 Operator Key: scope-85)
2014-06-12 00:33:24,653 [main] INFO org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher - Converting operator POLoad (Name: B: Load(hdfs://localhost:54310/input/data2:PigStorage(',')) - scope-86 Operator Key: scope-86)
2014-06-12 00:33:24,660 [main] INFO org.apache.spark.storage.MemoryStore - ensureFreeSpace(63445) called with curMem=317225, maxMem=559310438
2014-06-12 00:33:24,661 [main] INFO org.apache.spark.storage.MemoryStore - Block broadcast_5 stored as values to memory (estimated size 62.0 KB, free 533.0 MB)
2014-06-12 00:33:24,662 [main] INFO org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher - Converting operator POForEach (Name: B: New For Each(false,false)[bag] - scope-91 Operator Key: scope-91)
2014-06-12 00:33:24,663 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2043: Unexpected error during execution.
Details at logfile: /home/maheshk/Desktop/spork/spork/pig_1402512909094.log
Complete Log: https://gist.github.com/b895c51dbddfc19e0ab4
2014-06-12 00:33:24,608 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: SKEWED_JOIN
2014-06-12 00:33:24,609 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, DuplicateForEachColumnRewrite, GroupByConstParallelSetter, ImplicitSplitInserter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]}
2014-06-12 00:33:24,616 [main] INFO org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher - !!!!!!!!!! Launching Spark (woot) !!!!!!!!!!!!
2014-06-12 00:33:24,625 [main] INFO org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code.
2014-06-12 00:33:24,625 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-06-12 00:33:24,633 [main] INFO org.apache.pig.impl.builtin.PartitionSkewedKeys - input file: hdfs://localhost:54310/tmp/temp-1456742965/tmp-1456742965
2014-06-12 00:33:24,636 [main] INFO org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher - Converting operator POLoad (Name: A: Load(hdfs://localhost:54310/input/data1:PigStorage(',')) - scope-78 Operator Key: scope-78)
2014-06-12 00:33:24,650 [main] INFO org.apache.spark.storage.MemoryStore - ensureFreeSpace(63445) called with curMem=253780, maxMem=559310438
2014-06-12 00:33:24,650 [main] INFO org.apache.spark.storage.MemoryStore - Block broadcast_4 stored as values to memory (estimated size 62.0 KB, free 533.1 MB)
2014-06-12 00:33:24,652 [main] INFO org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher - Converting operator POForEach (Name: A: New For Each(false,false,false)[bag] - scope-85 Operator Key: scope-85)
2014-06-12 00:33:24,653 [main] INFO org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher - Converting operator POLoad (Name: B: Load(hdfs://localhost:54310/input/data2:PigStorage(',')) - scope-86 Operator Key: scope-86)
2014-06-12 00:33:24,660 [main] INFO org.apache.spark.storage.MemoryStore - ensureFreeSpace(63445) called with curMem=317225, maxMem=559310438
2014-06-12 00:33:24,661 [main] INFO org.apache.spark.storage.MemoryStore - Block broadcast_5 stored as values to memory (estimated size 62.0 KB, free 533.0 MB)
2014-06-12 00:33:24,662 [main] INFO org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher - Converting operator POForEach (Name: B: New For Each(false,false)[bag] - scope-91 Operator Key: scope-91)
2014-06-12 00:33:24,663 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2043: Unexpected error during execution.
Details at logfile: /home/maheshk/Desktop/spork/spork/pig_1402512909094.log
Complete Log: https://gist.github.com/b895c51dbddfc19e0ab4