AVRO-4197: Fix schema bytes defaults

[kellen]

What is the purpose of the change

AVRO-4197 Fixes incorrect handling of bytes in the context of schema production introduced in #2529 (AVRO-3876).

I am ambivalent about needing a special-case parameter inSchemaContext to revert back to the previous behavior. It seems like the basis for #2529 was that public json values (e.g. in metadata or via avro-tools) would more properly use the base64 encoding now provided rather than the previous ISO_8859_1 encoding. Since JacksonUtils is used in both situations, some kind of use-case specific switch is unfortunately required.

AVRO-3843 seems potentially related to this issue but I have not investigated.

Verifying this change

This change added tests and can be verified as follows:

• Added TestScheam.byteArrayDefaultField() test verifying that the default bytes values can round-trip

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

[kellen]

@martin-g Would you mind reapproving/retriggering the workflows?

[RyanSkraba]

Thanks for this contribution! This was definitely a behaviour change with unexpected consequences, and doesn't exist in Avro 1.11.5.

However, I can't seem to figure out what the PR that caused this change was fixing -- @kellen, do you think you could take another look at #2529 and see if reverting it would be a preferable solution to your issue?

[kellen]

Per the original PR "Current JacksonUtils is not symmetric"; I don't know if this was a "real problem" or a theoretical, annoyingly inconsistent behavior. I didn't dig too deeply into it but it seemed like the JacksonUtils.toJson may be used in other situations where you don't want this unusual byte encoding. @clesaec, can you shed some light on it?

[clesaec]

Hello,
The ancient version of JacksonUtils used TextNode to store an array of byte, that could modify the input with Charset considerations. Now, it is directly stored in BinaryNode, that directly store the array of bytes without any transformation.
Regards,
Christophe.

[kellen]

@clesaec That's the root of the current bug, yes, but I presume the encoding change to bytes was a desired outcome in some cases? Or was it just the result of trying to get a logical round-trip inside of jackson?

The side effect of your original change was to change the way schema defaults were encoded, which I would guess was not desired.

[RyanSkraba]

Hello @clesaec ! It's nice to hear from you! My apologies for not bringing you in sooner -- I thought you were away.

I created #3584 to investigate if we see any regressions in reverting to the 1.11.x behaviour -- it might be a good idea to reopen that JIRA and see if there's anything that needs to be fixed.

@kellen Let's make sure this gets fixed in the next minor release -- I'm working on it and it's in the list! Can you take a look at the revert PR and see if that's a solution that works for you?

[clesaec]

Hi @RyanSkraba ,
I'm indeed away, just few minutes from time to time to get avro news, but not a lot.
Yes, it may be useful to reopen the jira but I think I won't have time to look at it, or may be at Christmas, I have some days off.
Regards,
Christophe.

[kellen]

@RyanSkraba a revert would resolve the schema defaults issue, yes.

[RyanSkraba]

I reverted the change and I'll reopen the JIRA with a comment! Thanks for your contribution and your patience!

If you want to reopen, rebase on main and include the test case you wrote, I'd be happy to merge that!