The merge operation could fail when there's channel exception for coverage files

[hadi88]

We've observed that CoverageIdStrategy.obtainFirstId fails during the merge process with FileLockInterruptionException, ClosedByInterruptException, and other channels exceptions. Handling such exceptions could significantly improve robustness of the merge operation.

Unfortunately, I don't have a reproducer for this issue.

[fmeum]

It would be really helpful to have a reproducer for this, but I can see that being difficult to come up with.

We have observed rare inconsistencies in the shared coverage ID file in the past that resulted in a crash, but never managed to identify the problem. I haven't seen cases of these exceptions being thrown yet, but will take another look at the code and check whether we can handle them (more) gracefully.

[fmeum]

@hadi88 Do you have stack traces that you could share?

I can attempt a speculative fix that retries obtaining the lock in case of interrupts, but I would first like to confirm that this has a chance to fix the issue you are seeing.

[hadi88]

Initially, I observed a check failure in commitIdCount:

Caused by: java.lang.IllegalStateException: Check failed.
	at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.commitIdCount(CoverageIdStrategy.kt:175)
	... 34 more

This exception masks the root cause. When I changed the check failure into an if-condition, I observed the following two stack traces:

ERROR: Coverage IDs are out of sync	
com.code_intelligence.jazzer.agent.CoverageIdException: Failed to synchronize coverage IDs
	at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.obtainFirstId(CoverageIdStrategy.kt:167)
	at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.withIdForClass(CoverageIdStrategy.kt:90)
	at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.instrument(RuntimeInstrumentor.kt:215)
	at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.transformInternal(RuntimeInstrumentor.kt:185)
	at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.transform(RuntimeInstrumentor.kt:96)
	at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.transform(RuntimeInstrumentor.kt:161)
	at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
	at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1018)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:523)
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1018)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:523)
	at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
	at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3139)
	at java.base/java.lang.Class.getConstructor0(Class.java:3344)
	at java.base/java.lang.Class.getConstructor(Class.java:2151)
	at io.netty.channel.ReflectiveChannelFactory.<init>(ReflectiveChannelFactory.java:34)
	at io.netty.bootstrap.AbstractBootstrap.channel(AbstractBootstrap.java:110)
	...
Caused by: java.nio.channels.ClosedByInterruptException
	at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:199)
	at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:162)
	at java.base/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:228)
	at com.code_intelligence.jazzer.agent.CoverageIdStrategyKt.readFully(CoverageIdStrategy.kt:213)
	at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.obtainFirstId(CoverageIdStrategy.kt:123)
	... 43 more

ERROR: Coverage IDs are out of sync	
com.code_intelligence.jazzer.agent.CoverageIdException: Failed to synchronize coverage IDs
	at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.obtainFirstId(CoverageIdStrategy.kt:167)
	at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.withIdForClass(CoverageIdStrategy.kt:90)
	at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.instrument(RuntimeInstrumentor.kt:215)
	at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.transformInternal(RuntimeInstrumentor.kt:185)
	at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.transform(RuntimeInstrumentor.kt:96)
	at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.transform(RuntimeInstrumentor.kt:161)
	at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
	at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1018)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:523)
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1018)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:523)
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1018)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:523)
	...
Caused by: java.nio.channels.FileLockInterruptionException
	at java.base/sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1138)
	at java.base/java.nio.channels.FileChannel.lock(FileChannel.java:1065)
	at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.obtainFirstId(CoverageIdStrategy.kt:119)
	... 46 more

[fmeum]

It would be interesting to learn what causes these exceptions. Is your fuzz test multi-threaded? Does your JVM support concurrent class loading?

I am asking as I haven't seen these failures anywhere else yet and handling them properly isn't trivial. With more information, I might be able to come up with a reproducer.

@hadi88 - I wrote to you about the other issue, same here. Ping me so we can look at it more closely together?
david[dot]merian [at] code-intelligence[dot]com