From 74acaaa1c53b66c96d0645f8af7a76aab70df318 Mon Sep 17 00:00:00 2001 From: nick evans Date: Thu, 5 Feb 2026 17:25:20 -0500 Subject: [PATCH 1/2] =?UTF-8?q?=F0=9F=A5=85=20Add=20very=20basic=20backtra?= =?UTF-8?q?cking=20to=20the=20parser?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This can be used to catche parse errors, restore the parser state, and then either call a fallback or re-raise the error. NOTE: Reckless backtracking can lead to O(n**2) situations, so it should very rarely be used. Fallbacks should not backtrack. --- lib/net/imap/response_parser/parser_utils.rb | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/lib/net/imap/response_parser/parser_utils.rb b/lib/net/imap/response_parser/parser_utils.rb index 880d4cc2..99626d03 100644 --- a/lib/net/imap/response_parser/parser_utils.rb +++ b/lib/net/imap/response_parser/parser_utils.rb @@ -221,8 +221,14 @@ def exception(message) = ResponseParseError.new( message, parser_state:, parser_class: self.class ) - def current_state = [@lex_state, @pos, @token] - def parser_state = [@str, *current_state] + # This can be used to backtrack after a parse error, and re-attempt to + # parse using a fallback. + # + # NOTE: Reckless backtracking could lead to O(n²) situations, so this + # should very rarely be used. Ideally, fallbacks should not backtrack. + def restore_state(state) = (@lex_state, @pos, @token = state) + def current_state = [@lex_state, @pos, @token] + def parser_state = [@str, *current_state] end end From 7fb8b576f7530fe67588d751bcc035b6a32a495f Mon Sep 17 00:00:00 2001 From: nick evans Date: Tue, 3 Feb 2026 16:16:36 -0500 Subject: [PATCH 2/2] =?UTF-8?q?=F0=9F=90=9B=20Parse=20`resp-text`=20with?= =?UTF-8?q?=20invalid=20`resp-text-code`?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `resp-text` is defined as: ```abnf resp-text = ["[" resp-text-code "]" SP] text ; RFC3501 resp-text = ["[" resp-text-code "]" SP] [text] ; RFC9051 ``` And our even more lenient interpretation (which incorporates RFC9051 errata), is this: ```abnf resp-text = ["[" resp-text-code "]" [SP [text]] / [text] ``` And, although the `resp-text-code` grammar is elaborate, the final alternative is a superset of every other alternative (in both RFCs): ```abnf resp-text-code /= atom [SP 1*] ``` Notice that `text` is a superset of `atom`, `SP`, `"["`, `"]"`, and every other allowed character in `resp-text-code`. So, even if `resp-text-code` fails, `resp-text` may still succeed by parsing everything as `text`. However, (prior to this commit) the parser commits to `resp-text-code` as soon as `"["` is encountered at the beginning of `resp-text`. If what follows is not `resp-text-code`, maybe it doesn't begin with an `atom` or it doesn't have a closing `"]"`, then we fail to parse `resp-text` correctly. Fixing this requires either looking further ahead more than a single token or backtracking when `resp-text-code` fails to parse. In this case, I think backtracking is the best approach. We don't need to worry about backtracking taking exponential time, because the fallback (`text`) is exceptionally simple (does not call any other productions), will parse all the way up to the next CRLF, and so nothing backtrackable can be nested. This uses `ResponseParser#current_state` that was added for #599 and relies on #600 to ensure that debug error messages are not printed when backtracking. Fixes #597. --- lib/net/imap/response_parser.rb | 5 ++++ .../response_parser/quirky_behaviors.yml | 27 +++++++++++++++++++ 2 files changed, 32 insertions(+) diff --git a/lib/net/imap/response_parser.rb b/lib/net/imap/response_parser.rb index 189bc980..2c1d1dd1 100644 --- a/lib/net/imap/response_parser.rb +++ b/lib/net/imap/response_parser.rb @@ -1888,12 +1888,17 @@ def text? # We leniently re-interpret this as # resp-text = ["[" resp-text-code "]" [SP [text]] / [text] def resp_text + state = current_state if lbra? code = resp_text_code; rbra ResponseText.new(code, SP? && text? || "") else ResponseText.new(nil, text? || "") end + rescue ResponseParseError => error + raise if /\buid-set\b/i.match? error.message + restore_state state + text end # RFC3501 (See https://www.rfc-editor.org/errata/rfc3501): diff --git a/test/net/imap/fixtures/response_parser/quirky_behaviors.yml b/test/net/imap/fixtures/response_parser/quirky_behaviors.yml index e6bf9d48..f6f3e790 100644 --- a/test/net/imap/fixtures/response_parser/quirky_behaviors.yml +++ b/test/net/imap/fixtures/response_parser/quirky_behaviors.yml @@ -31,6 +31,33 @@ unparsed_data: "froopy snood" raw_data: "* 86 NOOP froopy snood\r\n" + "Microsoft Exchange issues an invalid resp-text-code": + comment: | + net-imap issue: https://github.com/ruby/net-imap/issues/597 + + Microsoft Exchange is issuing an invalid resp-text-code, but net-imap + should fallback to `resp-text = text` when resp-text-code fails to parse. + :debug: false # current version of backtracking still prints parse_errors + :response: "RUBY0001 OK [Error=\"Microsoft.Exchange.Data.Storage.WrongServerException: + Cross Server access is not allowed for mailbox xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\" + AuthResultFromPopImapEnd=0 Proxy=XXXXXXXXXXXXX.EURXXXX.PROD.OUTLOOK.COM:1993:SSL + MailboxBE=XXXXXXXXXXXXX.EURXXXX.PROD.OUTLOOK.COM + Service=Imap4] AUTHENTICATE completed.\r\n" + :expected: !ruby/struct:Net::IMAP::TaggedResponse + tag: RUBY0001 + name: OK + data: '[Error="Microsoft.Exchange.Data.Storage.WrongServerException: Cross Server + access is not allowed for mailbox xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" + AuthResultFromPopImapEnd=0 + Proxy=XXXXXXXXXXXXX.EURXXXX.PROD.OUTLOOK.COM:1993:SSL + MailboxBE=XXXXXXXXXXXXX.EURXXXX.PROD.OUTLOOK.COM Service=Imap4] + AUTHENTICATE completed.' + raw_data: "RUBY0001 OK [Error=\"Microsoft.Exchange.Data.Storage.WrongServerException: + Cross Server access is not allowed for mailbox xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\" + AuthResultFromPopImapEnd=0 Proxy=XXXXXXXXXXXXX.EURXXXX.PROD.OUTLOOK.COM:1993:SSL + MailboxBE=XXXXXXXXXXXXX.EURXXXX.PROD.OUTLOOK.COM Service=Imap4] AUTHENTICATE + completed.\r\n" + outlook.com puts an extra SP in ENVELOPE address lists: comment: | An annoying bug from outlook.com. They've had the bug for years, and