Skip to content

Enhance error handling in DelegatingMultiSinkOutputCommitter #1606

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
psainics wants to merge 1 commit into
base: develop
Choose a base branch
from
Open

Enhance error handling in DelegatingMultiSinkOutputCommitter #1606

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 27 additions & 6 deletions src/main/java/io/cdap/plugin/gcp/bigquery/sink/DelegatingMultiSinkOutputCommitter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@
import org.apache.hadoop.mapreduce.JobStatus;
import org.apache.hadoop.mapreduce.OutputCommitter;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.IOException;
import java.util.HashMap;
Expand All @@ -35,6 +37,7 @@
* Delegated instances are supplied along with a schema, which is used to configure the commit operation.
*/
public class DelegatingMultiSinkOutputCommitter extends OutputCommitter {
private static final Logger LOG = LoggerFactory.getLogger(DelegatingMultiSinkOutputCommitter.class);
private final Map<String, OutputCommitter> committerMap;
private final Map<String, Schema> schemaMap;
private final String projectName;
Expand Down Expand Up @@ -99,18 +102,28 @@ public boolean needsTaskCommit(TaskAttemptContext taskAttemptContext) throws IOE
@Override
public void commitTask(TaskAttemptContext taskAttemptContext) throws IOException {
for (String tableName : committerMap.keySet()) {
configureContext(taskAttemptContext, tableName);

committerMap.get(tableName).commitTask(taskAttemptContext);
try {
configureContext(taskAttemptContext, tableName);
committerMap.get(tableName).commitTask(taskAttemptContext);
} catch (IOException e) {
LOG.warn("BigQuery multi-sink table '{}' failed during task commit. Reason: {}",
tableName, getFailureReason(e), e);
throw e;
}
}
}

@Override
public void commitJob(JobContext jobContext) throws IOException {
for (String tableName : committerMap.keySet()) {
configureContext(jobContext, tableName);

committerMap.get(tableName).commitJob(jobContext);
try {
configureContext(jobContext, tableName);
committerMap.get(tableName).commitJob(jobContext);
} catch (IOException e) {
LOG.warn("BigQuery multi-sink table '{}' failed during job commit. Reason: {}",
tableName, getFailureReason(e), e);
throw e;
Copy link
Copy Markdown
Contributor

@itsankit-google itsankit-google Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logging and re-throwing an error is considered an anti-pattern.

Isn't this error propagated upwards and logged somewhere in pipeline run?

}
}
}

Expand Down Expand Up @@ -168,4 +181,12 @@ public void configureContext(JobContext context, String tableName) throws IOExce
gcsPath,
fields);
}

private String getFailureReason(IOException exception) {
Throwable rootCause = exception;
while (rootCause.getCause() != null) {
rootCause = rootCause.getCause();
}
return rootCause.getMessage() == null ? exception.getMessage() : rootCause.getMessage();
}
}
Loading