Skip to content

Add cache for single measurement validation#17899

Open
jt2594838 wants to merge 4 commits into
masterfrom
add_measurement_cache
Open

Add cache for single measurement validation#17899
jt2594838 wants to merge 4 commits into
masterfrom
add_measurement_cache

Conversation

@jt2594838

@jt2594838 jt2594838 commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Description

This PR adds a configurable cache for PathUtils#checkAndReturnSingleMeasurement so repeated measurement validation can reuse previous check results. The cache stores both valid and invalid validation results and can be disabled by setting the cache size to 0.

It also adds configuration plumbing for integration tests and an ignored performance IT that can be run manually to compare end-to-end write performance with the cache disabled and enabled.

Test Report

Local verification:

mvn spotless:apply -pl integration-test -P with-integration-tests
mvn spotless:apply -pl iotdb-core/datanode
mvn test-compile -DskipUTs -Drat.skip=true -pl integration-test -am -P with-integration-tests

Performance averages, 3 repeats per group, 80,000 rows per experiment:

measurementCount cacheSize Relation Cache Disabled Cache Enabled Enabled/Disabled
1000 500 cacheSize < measurementCount 4794 ms 4384 ms 0.914
2000 2000 cacheSize = measurementCount 9676 ms 8255 ms 0.853
4000 8000 cacheSize > measurementCount 19980 ms 17223 ms 0.862

@sonarqubecloud

Copy link
Copy Markdown

@codecov

codecov Bot commented Jun 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 51.35135% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 40.75%. Comparing base (1d8b23a) to head (d011d05).
⚠️ Report is 26 commits behind head on master.

Files with missing lines Patch % Lines
...java/org/apache/iotdb/commons/utils/PathUtils.java 21.73% 18 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #17899      +/-   ##
============================================
- Coverage     40.80%   40.75%   -0.06%     
- Complexity     2612     2621       +9     
============================================
  Files          5200     5244      +44     
  Lines        353687   362667    +8980     
  Branches      45252    46685    +1433     
============================================
+ Hits         144326   147801    +3475     
- Misses       209361   214866    +5505     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Caideyipi

Copy link
Copy Markdown
Collaborator

I think this part has a correctness issue: checkTsFileResourceExists currently warns for every .tsfile because it only checks the filename suffix and never checks whether the corresponding resource file exists. For a normal snapshot containing both xxx.tsfile and xxx.tsfile.resource, this still logs The associated resource file ... is not found in the snapshot, which is a false warning. Please check
ew File(file.getAbsolutePath() + TsFileResource.RESOURCE_SUFFIX).exists() before warning, or remove this unrelated snapshot change from this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants