[SPARK-50886][BUILD][3.5] Upgrade Avro to 1.11.4

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to upgrade Avro to 1.11.4.

Why are the changes needed?

To bring the latest bug fixes like the following.

• https://github.com/apache/avro/releases/tag/release-1.11.3
  • AVRO-3819: Centralize system properties that limit allocations avro#2432
  • AVRO-3789: [java] fix map comparison in GenericData avro#2318
  • AVRO-2771: Refactor custom codable check avro#1720
• https://github.com/apache/avro/releases/tag/release-1.11.4
  • AVRO-3985: Add trusted packages support in SpecificData avro#2980

Does this PR introduce any user-facing change?

Yes, this is a dependency change.

How was this patch tested?

Pass the CIs.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

cc @LuciferYang

[dongjoon-hyun]

Could you review this PR, @panbingkun ?

[LuciferYang]

LGTM

[LuciferYang]

[error] 
[error]   bad constant pool index: 0 at pos: 3287
[error]      while compiling: /home/runner/work/spark/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
[error]         during phase: globalPhase=typer, enteringPhase=namer
[error]      library version: version 2.13.8
[error]     compiler version: version 2.13.8
[error]   reconstructed args: -bootclasspath /usr/lib/jvm/temurin-8-jdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/temurin-8-jdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/temurin-8-jdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/temurin-8-jdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/temurin-8-jdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/temurin-8-jdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/temurin-8-jdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/temurin-8-jdk-amd64/jre/classes:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scala-lang/scala-library/2.13.8/scala-library-2.13.8.jar -sourcepath /home/runner/work/spark/spark -explaintypes -feature -unchecked -Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBaseOps.scala:s -Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBase.scala:s -Wconf:msg=^(?=.*?a value of type)(?=.*?cannot also be).+$:s -Wconf:cat=unchecked&msg=eliminated by erasure:s -Wconf:cat=unchecked&msg=outer reference:s -Wconf:cat=deprecation&msg=procedure syntax is deprecated:e -Wconf:msg=method without a parameter list overrides a method with a single empty one:s -Wconf:msg=method with a single empty parameter list overrides method without any parameter list:s -Wconf:msg=Auto-application to \`\(\)\` is deprecated:s -Wconf:msg=^(?=.*?Widening conversion from)(?=.*?is deprecated because it loses precision).+$:s -Wconf:msg=^(?=.*?method|value|type|object|trait|inheritance)(?=.*?deprecated)(?=.*?since 2.13).+$:s -Wconf:cat=other-pure-statement&site=org.apache.spark.sql.streaming.sources.StreamingDataSourceV2Suite.testPositiveCase.\$anonfun:wv -Wconf:cat=other-pure-statement&site=org.apache.spark.scheduler.OutputCommitCoordinatorSuite.<local OutputCommitCoordinatorSuite>.futureAction:wv -Wconf:cat=other-pure-statement&site=org.apache.spark.streaming.util.FileBasedWriteAheadLog.readAll.readFile:wv -Wconf:cat=other-match-

The compilation for Scala 2.13 failed. Can we re-trigger it to confirm if it was just a random occurrence?

The compilation for Scala 2.13 failed.

[LuciferYang]

I tested it locally and encountered the same error.

[dongjoon-hyun]

Oh.. Let me check.

[LuciferYang]

I tried it out and found that 1.11.3 can be compiled successfully. The issue likely stems from 1.11.4.

[dongjoon-hyun]

Oh, surprising.

Actually, 1.11.4 was requested last year.

• https://lists.apache.org/thread/1w9nmrhy716hl6fz6w3pw9w37tv6o0nl

Let me dig more.

[dongjoon-hyun]

The root cause seems due to the SBT dependency bug. I pinned commons-compress version in SparkBuild.scala and verified Scala 2.13 compilation. Let's see the CI result, @LuciferYang . Thank you!

$ dev/change-scala-version.sh 2.13
$ build/sbt package -Pscala-2.13
...
[success] Total time: 88 s (01:28), completed Jan 18, 2025, 7:41:30 PM

[dongjoon-hyun]

BTW, this patch and the pinning is only for branch-3.5 because branch-3.5 has old Scala version, 2.13.8.

[LuciferYang]

It seems so. I tried compiling with Maven, and it worked fine at commit 86dfeae. I also tried compiling the latest commit using sbt, and it worked as well.

[LuciferYang]

+1, LGTM
pending tests

[dongjoon-hyun]

Thank you always for your deep review and co-investigation!

[dongjoon-hyun]

All tests passed. Merged to branch-3.5.

[panbingkun]

+1, late LGTM