fix: false-positive password detection for PDFs whose /ID starts with UTF-16 BOM bytes

[tegaryas]

Summary

Two related PDF loading bugs fixed across syncfusion_flutter_pdf and syncfusion_flutter_pdfviewer (v33.1.44):

1. False-positive password detection — PDFs with no user password were incorrectly treated as password-protected when the file's /ID entry starts with 0xFE 0xFF.
2. Password-protected PDF fails to render after correct password is entered — opening a genuinely password-protected PDF in SfPdfViewer showed the password dialog correctly, but the document stayed blank after entering the right password.

---

Problem 1 — False-positive password detection

Opening certain PDFs that require no password caused Syncfusion to show the password dialog or throw:

Invalid argument (password): Cannot open an encrypted document. The password is invalid.: ""

The same files opened without issue in viewers backed by PDFium or CGPDFDocument (e.g. pdfx).

Root Cause

PdfString (hex-string constructor path) checks whether decoded bytes start with 0xFE 0xFF (UTF-16 Big-Endian BOM). When triggered it decoded the bytes as UTF-16 text but first reset data to an empty list, discarding the original raw bytes and replacing them with only the low-8-bits of each decoded Unicode character:

/ID [ <FEFF6A5543956412A465BA1C426B5263> ... ]
       ^^^^
       Looks like UTF-16 BOM → triggered the bug

	Bytes
Correct 16 bytes	FE FF 6A 55 43 95 64 12 A4 65 BA 1C 42 6B 52 63
After bug (7 bytes)	55 95 12 65 1C 6B 63

The /ID entry feeds directly into the PDF encryption key derivation (PDF spec §3.5, Algorithm 2). With only 7 wrong bytes instead of the original 16, the computed RC4 key (b3242cdf73) differed from the correct one (88b6e1d1af), so the computed U value never matched the stored U entry and authentication failed for every document whose ID starts with these two bytes.

A thorough trace of the entire syncfusion_flutter_pdf library confirmed that this is the only location where data is mutated after a BOM detection on raw bytes. All other BOM checks in the library either operate on local variables, skip BOM bytes by advancing a range pointer, or check already-decoded strings — none mutate the field-level data array.

---

Problem 2 — Password-protected PDF blank after correct password

For a genuinely password-protected PDF, the password dialog appeared correctly, but after the user entered the correct password the preview stayed blank and onDocumentLoadFailed fired with "There was an error opening this document.".

Root Cause

_renderDigitalSignatures() opens a second PdfDocument from _pdfBytes to inspect form fields and flatten any digital signatures before passing bytes to the native renderer. It did so without forwarding the password:

// before fix
final PdfDocument pdfDocument = PdfDocument(inputBytes: _pdfBytes);

For an encrypted document this threw ArgumentError: Cannot open an encrypted document, which propagated unhandled to the outer catch and surfaced as the generic load error instead of rendering the document.

Additionally, _renderDigitalSignatures() was called inline without any null guard or error isolation — any exception from form-field processing aborted the entire load.

---

Changes

syncfusion_flutter_pdf — pdf_string.dart

Skip the low-byte append loop for 16-byte hex strings. PDF file identifiers (/ID) are always 16-byte binary values — appending decoded character bytes corrupts the raw content used in encryption key derivation. The data = <int>[] reset was already removed; this adds the length guard to fully protect all 16-byte binary fields.

  if (data![0] == 0xfe && data![1] == 0xff) {
    this.value = decodeBigEndian(data, 2, data!.length - 2);
    isHex = false;
-   data = <int>[];
-   for (int i = 0; i < this.value!.length; i++) {
-     data!.add(this.value!.codeUnitAt(i).toUnsigned(8));
-   }
+   // 16-byte fields (e.g. /ID) are raw binary identifiers — never
+   // append decoded character bytes to them.
+   if (data!.length != 16) {
+     for (int i = 0; i < this.value!.length; i++) {
+       data!.add(this.value!.codeUnitAt(i).toUnsigned(8));
+     }
+   }
  }

syncfusion_flutter_pdfviewer — pdfviewer.dart

1. Pass password to _renderDigitalSignatures()

- final PdfDocument pdfDocument = PdfDocument(inputBytes: _pdfBytes);
+ final PdfDocument pdfDocument = PdfDocument(
+   inputBytes: _pdfBytes,
+   password: _password,
+ );

2. Isolate _renderDigitalSignatures() with null guard and try/catch

Digital signature flattening is best-effort. Any failure now falls back to passing _pdfBytes directly to the native renderer instead of aborting the entire document load.

+ Uint8List? signatureBytes;
+ if (_document != null) {
+   try {
+     signatureBytes = _renderDigitalSignatures();
+   } catch (_) {
+     // Digital-signature flattening is best-effort; fall back to raw bytes.
+   }
+ }
  final int pageCount = await _plugin.initializePdfRenderer(
-   _renderDigitalSignatures() ?? _pdfBytes,
+   signatureBytes ?? _pdfBytes,
    _password,
  );

3. Suppress spurious onDocumentLoadFailed when password dialog is shown

onDocumentLoadFailed("Empty Password Error") was previously fired even when canShowPasswordDialog: true — i.e., when the built-in dialog was about to appear. This caused host apps to surface a confusing error to users. The callback is now only fired when canShowPasswordDialog is false (no dialog to handle it).

  if (widget.password == '' || widget.password == null) {
-   widget.onDocumentLoadFailed!(...'Empty Password Error'...);
+   if (!widget.canShowPasswordDialog) {
+     widget.onDocumentLoadFailed!(...'Empty Password Error'...);
+   }
  }

---

Test Plan

• PDF with /ID starting 0xFE 0xFF → opens without any password prompt
• Truly password-protected PDF, no widget.password → password dialog appears, no onDocumentLoadFailed callback fires
• Enter correct password in dialog → document renders correctly
• Enter wrong password in dialog → "Invalid Password" error shown, dialog remains
• Normal unencrypted PDF → opens normally, no prompt
• Password-protected PDF with digital signature fields → opens after entering password, signatures preserved
• Call PdfDocument(inputBytes: bytes) directly on an empty-user-password PDF → no longer throws

[AbinayaSF4962]

Hi @tegaryas ,

Thank you for sharing your proposal and patch to fix incorrect password detection in non-protected PDFs (due to /ID starting with 0xFE 0xFF) and the rendering issue in SfPdfViewer after entering the correct password.

We are currently reviewing the proposed change and will investigate this issue further. Could you please share the document where the problem occurred, along with the stack trace details?

Please note that, although the Flutter PDF library and PdfViewer source codes are publicly available on GitHub for transparency, we do not accept direct contributions to the repository. This policy ensures consistency and quality across all releases. Nevertheless, your feedback and suggestions are highly valued, and we encourage you to continue sharing ideas through our official support channels.

Thank you for your understanding and continued support.

Regards,
Abinaya E