Skip to content

Fix silent deserialization failure with unique together constraint#313

Open
ozer550 wants to merge 4 commits intolearningequality:release-v0.8.xfrom
ozer550:fix-silent-deserialization-failure-with-unique-together-constraint
Open

Fix silent deserialization failure with unique together constraint#313
ozer550 wants to merge 4 commits intolearningequality:release-v0.8.xfrom
ozer550:fix-silent-deserialization-failure-with-unique-together-constraint

Conversation

@ozer550
Copy link
Copy Markdown
Member

@ozer550 ozer550 commented Apr 2, 2026

Summary

  • Added IntegrityError to the caught exceptions in the deserialization pipeline so unique constraint violations are handled instead of silently ignored.
  • Replaced the bulk upsert in the non-self-referential branch with per-record saves.
  • Added regression tests in test_controller.py for the unique constraint scenario.

TODO

  • Have tests been written for the new code?
  • Has documentation been written/updated?
  • New dependencies (if any) added to requirements file

Reviewer guidance

  • Check if the regression tests make sense!
  • Manually syncing two devices with voilating unique constraint for fields.

Issues addressed

closes #306

:returns: True if save succeeded, False otherwise
"""
try:
with transaction.atomic():
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is so that the outer transaction stays healthy, and we can keep saving other records.

Copy link
Copy Markdown
Member

@bjester bjester left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really mostly complete, just some cleanup and tidying code.

And the small deserialization_exception enhancement!

exceptions.ValidationError,
exceptions.ObjectDoesNotExist,
ValueError,
) as e:
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we don't need this try/except anymore? So we can simplify this, leveraging the behavior of _save_deserialized_record

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I initially tried removing the try/except around _deserialize_store_model to reduce code, but I think keeping it is actually better. The reason is that there are two different things that can go wrong at two different moments. the json from the Store might fail having a deserialization failure. Second, our unique constraint violation case. clarity is worth the few extra lines?

continue
_save_deserialized_record(
store_model, app_model, model.__name__, excluded_list
)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The excluded_list doesn't seem necessary here, so maybe it's made optional for this function?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do execute excluded_list.append(store_model.id) to keep track of shared failed ids?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes but the purpose for that list is this very loop. So modifying the list is a bit redundant, and theoretically makes the in check slower every iteration without any benefit.

The store_model.id is unique and the same as app_model.id, so as this loops through the store+app models, if the ID gets added to the excluded_list, it will never come up again in the loop.


:returns: True if save succeeded, False otherwise
"""
from django.db import transaction as django_transaction
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strange, I did see the issue the CI. We already have transaction imported. Once the other stuff is tackled, it might be worth dropping the commit to see if it fails again.

@ozer550 ozer550 marked this pull request as ready for review April 6, 2026 15:57
@ozer550 ozer550 changed the base branch from release-v0.9.x to release-v0.8.x April 6, 2026 15:59
@bjester bjester self-assigned this Apr 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants