[interop] Cannot convert a Ruby String with BINARY encoding containing non-US-ASCII character 226 to a Java String

[fniephaus]

Repro

GraalVM MultiLanguage Shell 19.0.0
Copyright (c) 2013-2019, Oracle and/or its affiliates
  JavaScript version 19.0.0
  Ruby version 2.6.2
js> ruby>
ruby> Polyglot.export('string', 226.chr)
Internal error occured: org.graalvm.polyglot.PolyglotException: org.truffleruby.language.control.RaiseException: Cannot convert a Ruby String with BINARY encoding containing non-US-ASCII character 226 to a Java String (CannotConvertBinaryRubyStringToJavaString)
	from org.truffleruby.core.rope.RopeOperations.decodeNonAscii(RopeOperations.java:167)
	from org.truffleruby.core.rope.RopeOperations.decodeRopeSegment(RopeOperations.java:223)
	from org.truffleruby.core.rope.RopeOperations.decodeRopeSegment(RopeOperations.java:216)
	from org.truffleruby.core.rope.RopeOperations.decodeRope(RopeOperations.java:212)
	from org.truffleruby.core.string.StringOperations.getString(StringOperations.java:60)
	from org.truffleruby.interop.ToJavaStringNodeGen.executeAndSpecialize(ToJavaStringNodeGen.java:186)
	from org.truffleruby.interop.ToJavaStringNodeGen.executeToJavaString(ToJavaStringNodeGen.java:96)
	from org.truffleruby.interop.RubyToForeignNode.convertString(RubyToForeignNode.java:31)
	from org.truffleruby.interop.RubyToForeignNodeGen.executeAndSpecialize(RubyToForeignNodeGen.java:56)
	from org.truffleruby.interop.RubyToForeignNodeGen.executeConvert(RubyToForeignNodeGen.java:40)
Translated to internal error (RuntimeError)
Run with --verbose to see the full stack trace.

Expected Behavior

GraalVM MultiLanguage Shell 19.0.0
Copyright (c) 2013-2019, Oracle and/or its affiliates
  JavaScript version 19.0.0
  Ruby version 2.6.2
js> Polyglot.export('string', String.fromCharCode(226));
â

[chrisseaton]

BINARY is really ASCII-8BIT, but which interpretation of the 8th bit do you want?

According to Wikipedia, 226 in various 8 bit vaiants could mean any of âтقβגโā. How do we pick?

[chrisseaton]

MRI doesn't know what you want either:

irb(main):001:0> 226.chr
=> "\xE2"

[eregon]

I think the solution here is to just pass an Encoding to Integer#chr:

226.chr(Encoding::UTF_8)

Then it should work fine to export it.
@fniephaus Could you try that?

For bytes below 128, Integer#chr uses US-ASCII, but above it uses the BINARY encoding by default, so an explicit encoding needs to be specified in that case.

[eregon]

Yes, that works in GraalVM:

ruby> Polyglot.export('string', 226.chr(Encoding::UTF_8))
"â"

@fniephaus Is that OK for your use case?

[fniephaus]

Our problem occurs when parsing a rather long text file (118MB) and causes an error. The workaround should do for now, thanks! Nonetheless, the example also shows that TruffleRuby violates the interop contract: interop's isString on 226.chr returns true and then asString fails. I don't know what the correct behavior is in terms of interop, maybe @chumer could clarify that? In particular, what encoding should interop strings have? I think utf-something would be a reasonable choice.

[chrisseaton]

Java strings don't have an encoding - that's an implementation detail. They do have a character set, which is Unicode.

I see what you mean about the interop contract though - we probably should not return isString if we can't convert.

Maybe we should insert the replacement character (�) when we are asked to covert character that we can't understand?

[eregon]

Our problem occurs when parsing a rather long text file (118MB) and causes an error.

The encoding of the String should then be set as non-binary (e.g., File.read instead of File.binread), otherwise it's not representable as a Java String, only as a byte[].

Maybe we should insert the replacement character (�) when we are asked to covert character that we can't understand?

I think almost nobody wants that behavior, here the error reveals a problem when reading the file, so I argue this is behavior as intended.

[chrisseaton]

What about the interop contract though? Maybe binary strings should not be isString?

[eregon]

@chumer Is it a violation if asString throws?

[fniephaus]

@eregon IIRC there are assertions for that in TCK. Do you test TruffleRuby against TCK? If isString returns true, asString shouldn't fail.

[eregon]

@fniephaus We do test against the TCK, but probably we don't have an example for a binary Ruby String with non-7-bit characters.
Making binary Ruby String with non-7-bit characters not isString would make some sense, but that sounds confusing for users. Maybe making all binary Ruby String non-isString would be easier to understand.

[fniephaus]

what about converting binary strings to byte[] and return a new Java string for that array? I think it’s also confusion that binary strings cannot be converted to char*/byte[] at all. It might not always makes sense but that’s the user’s problem maybe?