Add `TimestampWithOffset` canonical extension type by serramatutu · Pull Request #558 · apache/arrow-go

serramatutu · 2025-10-30T11:16:25Z

Which issue does this PR close?

This PR implements the new arrow.timestamp_with_offset canonical extension type for arrow-go.

Rationale for this change

Be compatible with Arrow spec.

What changes are included in this PR?

This commit adds a new TimestampWithOffset extension type. This type represents a timestamp column that stores potentially different timezone offsets per value. The timestamp is stored in UTC alongside the original timezone offset in minutes. The offset in minutes can be primitive encoded, dictionary encoded or run-end encoded.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes, this is a new canonical extension type.

…48002) ### Rationale for this change Closes #44248 Arrow has no built-in canonical way of representing the `TIMESTAMP WITH TIME ZONE` SQL type, which is present across multiple different database systems. Not having a native way to represent this forces users to either convert to UTC and drop the time zone, which may have correctness implications, or use bespoke workarounds. A new `arrow.timestamp_with_offset` extension type would introduce a standard canonical way of representing that information. Rust implementation: apache/arrow-rs#8743 Go implementation: apache/arrow-go#558 [DISCUSS] [thread in the mailing list](https://lists.apache.org/thread/yhbr3rj9l59yoxv92o2s6dqlop16sfnk). ### What changes are included in this PR? Proposal and documentation for `arrow.timestamp_with_offset` canonical extension type. ### Are these changes tested? N/A ### Are there any user-facing changes? Yes, this is an extension to the arrow format. * GitHub Issue: #44248 --------- Co-authored-by: David Li <li.davidm96@gmail.com> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Co-authored-by: Felipe Oliveira Carvalho <felipekde@gmail.com>

zeroshade · 2025-12-22T18:05:19Z

+				timestamps.UnsafeAppendBoolToBitmap(false)
+				offsets.UnsafeAppendBoolToBitmap(false)


same as above, these are non-nullable according to the spec.

I did change this but now the IPC roundtrip tests are failing because the internal count of nulls is different between the thing before IPC and after IPC.

To make them pass we'll need to relax the arrow.RecordEqual() to ignore NullN() when the field is not nullable. Alternatively we can change it so that an array builder sets nulls = 0 if the array is not nullable, regardless of what has been set before in calls to UnsafeAppendBoolToBitmap() etc

Related: https://lists.apache.org/thread/7gbqjwykh1ob3xbvwph3ljsdl5c7kxpd

zeroshade · 2026-02-05T23:25:44Z

+	*arrow.Int8Type | *arrow.Int16Type | *arrow.Int32Type | *arrow.Int64Type |
+		*arrow.Uint8Type | *arrow.Uint16Type | *arrow.Uint32Type | *arrow.Uint64Type


this and the DictIndexType are both identical, instead of duplicating this we should either create a single type constraint or embed one in the other:

type TimestampWithOffsetRunEndsType interface { DictIndexType }

My personal preference would be to have a single type though, but I'm not averse to having the two separate ones if necessary.

Fixed: 231cb1e

Actually I had to revert. Spec says those types are different:

Dict index is either signed or unsigned int of 8-64 bits

The index type of a Dictionary type can only be an integer type, preferably signed, with width 8 to 64 bits.

Run ends is a signed int of 16-64 bits

The run end type of a Run-End Encoded type can only be a signed integer type with width 16 to 64 bits.

https://arrow.apache.org/docs/format/Columnar.html#data-types

This commit adds a new `TimestampWithOffset` extension type that can be used to represent timestamps with per-row timezone information. It stores information in a `struct` with 2 fields, `timestamp=[T, "UTC"]`, where `T` can be any `arrow.TimeUnit` and `offset_minutes=int16`, which represents the offset in minutes from the UTC timestamp.

This commit allows `TimestampWithOffset` to be dict-encoded. - I made `NewTimestampWithOffsetType` take in an input `offsetType arrow.DataType`. It returns an error if the data type is not valid. - I added a new infallible `NewTimestampWithOffsetTypePrimitiveEncoded` to make the encoding explicit. - I added `NewTimestampWithOffsetTypeDictionaryEncoded` which returns an error in case the given type is not a valid dictionary key type. - I made all tests run in a for loop with all possible allowed encoding types, ensuring all encodings work.

Smartly iterate over offsets if they're run-end encoded instead of doing a binary search at every iteration. This makes the loops O(n) instead of O(n*logn).

Changed a lot of things based on Matt's suggestions.

Stop using `time=0` as a sentinel for null in `iterValues()`. Instead, return a tuple of `(time.Time, bool)` where the bool indicates whether the value is valid or not. Added a test for it.

serramatutu mentioned this pull request Oct 30, 2025

GH-44248: [Format] Add TimestampWithOffset canonical extension type apache/arrow#48002

Merged

felipecrv self-requested a review November 1, 2025 01:55

serramatutu force-pushed the serramatutu/TimestampWithOffset/go branch 3 times, most recently from 95230ad to b9b8bf2 Compare December 22, 2025 14:33

serramatutu changed the title ~~[DRAFT] Add TimestampWithOffset extension type~~ Add TimestampWithOffset extension type Dec 22, 2025

serramatutu marked this pull request as ready for review December 22, 2025 14:42

serramatutu requested a review from zeroshade as a code owner December 22, 2025 14:42

serramatutu changed the title ~~Add TimestampWithOffset extension type~~ Add TimestampWithOffset canonical extension type Dec 22, 2025

serramatutu force-pushed the serramatutu/TimestampWithOffset/go branch from b9b8bf2 to ccdd288 Compare December 22, 2025 15:17

zeroshade requested changes Dec 22, 2025

View reviewed changes

serramatutu force-pushed the serramatutu/TimestampWithOffset/go branch from ccdd288 to cee17a3 Compare January 30, 2026 12:21

serramatutu requested a review from zeroshade February 2, 2026 13:29

zeroshade requested changes Feb 5, 2026

View reviewed changes

serramatutu force-pushed the serramatutu/TimestampWithOffset/go branch from 2762d64 to 0249806 Compare April 10, 2026 21:56

serramatutu added 14 commits April 15, 2026 11:01

Allow run-end encoding of TimestampWithOffset

1abfe81

Optimize Values() and MarshalJSON() for REE

d95f78a

Smartly iterate over offsets if they're run-end encoded instead of doing a binary search at every iteration. This makes the loops O(n) instead of O(n*logn).

General code smell patches

5d97f9c

Changed a lot of things based on Matt's suggestions.

Test with -08:30 and 11:00 timezone

7c9b218

Rename constructors

5ce8c0b

Leverage generics for REE and dict-encoding

a3aca94

Make equals also check for data type

db0b04d

Remove outdated docs

ee12ad0

Remove unnecessary IsInteger call

bb210f9

Remove specific type for TimestampWithOffsetRunEndsType

9292026

Uncomment test

c5764fe

Fix REE ends type

aa92484

serramatutu added 2 commits April 15, 2026 12:12

Support unix epoch

d955cdb

Stop using `time=0` as a sentinel for null in `iterValues()`. Instead, return a tuple of `(time.Time, bool)` where the bool indicates whether the value is valid or not. Added a test for it.

Fix inconsistency with UnsafeAppend()

8383063

serramatutu force-pushed the serramatutu/TimestampWithOffset/go branch from 0249806 to bc577e9 Compare April 15, 2026 10:35

serramatutu added 2 commits April 15, 2026 13:03

Stop appending nulls to child arrays

9a5bd6c

Fix lint

8a7d6d7

serramatutu force-pushed the serramatutu/TimestampWithOffset/go branch from bc577e9 to 8a7d6d7 Compare April 15, 2026 11:05

Simplify fieldValuesFromTime() implementation

3a9a8f2

		timestamps.UnsafeAppendBoolToBitmap(false)
		offsets.UnsafeAppendBoolToBitmap(false)

		arrow.Int8Type \| arrow.Int16Type \| arrow.Int32Type \| arrow.Int64Type \|
		arrow.Uint8Type \| arrow.Uint16Type \| arrow.Uint32Type \| arrow.Uint64Type

Conversation

serramatutu commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zeroshade Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

serramatutu Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zeroshade Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

serramatutu Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

serramatutu Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

serramatutu commented Oct 30, 2025 •

edited

Loading

serramatutu Apr 15, 2026 •

edited

Loading

serramatutu Apr 15, 2026 •

edited

Loading