Passing `null` values into transformers

[tovbinm]

Howdy,

I was trying to run a StringIndexer model with setHandleInvalid("keep") on input { "inputCol" : null } but I got this error:

org.apache.avro.AvroTypeException: Expected string. Got VALUE_NULL

	at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:698)
	at org.apache.avro.io.JsonDecoder.readString(JsonDecoder.java:227)
	at org.apache.avro.io.JsonDecoder.readString(JsonDecoder.java:214)
	at org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:201)
	at com.opendatagroup.hadrian.data.PFADatumReader.readString(data.scala:582)
	at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:422)
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:180)
	at org.apache.avro.specific.SpecificDatumReader.readField(SpecificDatumReader.java:116)
	at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:230)
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:174)
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152)
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:144)
	at com.opendatagroup.hadrian.jvmcompiler.PFAEngineBase.fromJson(jvmcompiler.scala:1276)
	at com.opendatagroup.hadrian.jvmcompiler.PFAEngineBase.jsonInput(jvmcompiler.scala:1459)

How do I properly pass nulls into it?

[tovbinm]

Digging a bit deeper: it seems that StringIndexer allows nullable string inputs, while aardpfark does not.

[MLnick]

Currently we haven’t gotten to supporting it in the stringindexer. It is possible though

…

On Mon, 27 Aug 2018 at 18:41, Matthew Tovbin ***@***.***> wrote: Digging a bit deeper: it seems that StringIndexer allows nullable string inputs, while aardpfark does not <https://github.com/CODAIT/aardpfark/blob/master/src/main/scala/com/ibm/aardpfark/spark/ml/feature/StringIndexerModel.scala#L28> . — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#12 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA_SBwfxBThHggBlikEi0joy0YiCStESks5uVCFOgaJpZM4WOFyL> .

[tovbinm]

In general most of the spark stages support nulls, so I think this issue will come up again.