Extremely slow validation on repeated field with items validation

[andrey-filenko-quarks-tech]

Description

When I am using validation rules on items for repeated fields with env variable PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=upb, validation is extremely slow

Steps to Reproduce

Using the following message:

message RepeatedItems {
  repeated int64 items = 1 [
    (buf.validate.field).repeated = {
      items: {
        int64: {gt: 0}
      }
    }
  ];
}

Run this code:

import protovalidate
import time
from gen.tests.example.v1 import validations_pb2

msg = validations_pb2.RepeatedItems(items=[i + 1 for i in range(2000)])

start = time.perf_counter()
protovalidate.validate(msg)
end = time.perf_counter()
print(f"Finished one in {end-start:.2f} seconds")

Is executing about:

PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=upb python main.py
Finished one in 2.08 seconds

and with PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python:

PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python python main.py
Finished one in 0.01 seconds

But I guess both are pretty slow for just a validation call.

I have tried pre-warming up and doing it in a loop, however, numbers are the same after first call.

Environment

• Operating System: macOS Sequoia
• Compiler/Toolchain: Apple clang version 17.0.0 (clang-1700.0.13.3)
• Protobuf Compiler & Version: buf v1.54.0
• Protovalidate Version: v0.8.0

[smaye81]

Hi @andrey-filenko-quarks-tech can you provide a bit more info:

• Python version
• Protobuf version
• buf.gen.yaml (if not using Buf, show protoc command for generating code)

I was able to recreate this on my machine (Apple M1). However, I'm not seeing the 0.01s result for using the python backend. I actually see comparable (albeit slow) results for both:

$ PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=upb python3 main.py
Finished one in 3.83 seconds

$ PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python python3 main.py
Finished one in 3.92 second

Definitely something we should take a look at, but want to gather all the information first.

[andrey-filenko-quarks-tech]

Hi @smaye81

1. Python 3.11.5
2. Protobuf Python Version: 6.30.1
3. I have provided an example with slow validation by generating it in this pull request, so, I guess buf.gen.yaml is the same as in main branch of the current repository

[stefanvanburen]

hi @andrey-filenko-quarks-tech, thanks for the report.

Took a little peek this morning (you can see my terrible 'printf' style debugging in main...svanburen/perf).

Unfortunately, it looks like we're spending nearly all of our time evaluating the CEL expression, which we need to run 10,000 (2,000 * 5 rules) times because of the multiple rules associated with lt in protovalidate.

I think most of the perf gains here are probably from improvements in the underlying cel-python implementation (they're aware that things are slow). Unfortunately, we're also stuck (for now) on cel-python@0.2.0.

The only thing I can really think of here for protovalidate-python to look to improve is if we're able to switch from an InterpretedRunner to a CompiledRunner for repeated fields, but we're already monkey patching the underlying InterpretedRunner and I'm not sure we'd be able to do the same for the CompiledRunner. I haven't taken a look at if that's even a viable refactor.

Given that, for now I'm going to close this out, as I'm hoping we get perf improvements under the hood from eventually being able to upgrade to newer versions of cel-python. However, open to specific suggestions or contributions that improve the performance of protovalidate-python. Thanks again for the report!