Log undecodable CAN frames to per-ID CSVs under unknown_ids and document E2E procedure

[Copilot]

This change extends telemetry output to capture CAN frames that do not decode to any configured signal, without changing existing known-signal CSV behavior. Unknown frames are now persisted as per-ID CSVs in a dedicated subdirectory under the configured output path.

• CAN read/decode path

  • Track whether a received frame decoded at least one non-placeholder signal.
  • If no decode succeeds, route the raw frame to unknown-ID logging.

• Writer enhancements

  • Create <outdir>/unknown_ids during writer initialization.
  • Add writer_append_unknown(...) to append unknown frames to one file per CAN ID:
    • Path format: <outdir>/unknown_ids/%08X.csv (e.g., 00000777.csv)
    • CSV schema: timestamp_ns,dlc,data_hex
  • Keep existing per-signal writer behavior unchanged.

• Documentation / E2E procedure

  • Update README smoke-test flow to include an unknown-ID frame.
  • Add a concise end-to-end verification procedure for:
    • known ID → per-signal CSVs
    • unknown ID → per-ID CSV under unknown_ids
    • repeated unknown ID frames appending to the same file.

bool decoded_any = false;
/* ... iterate matching signals, decode ... */
if (decoder_extract(&node->sig, frame.data, frame.can_dlc, &dv) == 0) {
    decoded_any = true;
    writer_append(w, &node->sig, &dv);
}
if (!decoded_any) {
    writer_append_unknown(w, id, frame.data, frame.can_dlc);
}

Original prompt

Implement logging of unknown CAN IDs to per-ID CSV files under a subdirectory, and add an end-to-end test procedure.

Repository: sbasu107/can-telem-cloud

Goal

• In addition to the existing behavior (decode known signals defined in format.json and append to per-signal CSVs), also log raw CAN frames whose CAN ID does not successfully decode to any signal definition.
• Create one CSV per unknown CAN ID.
• Store these unknown-ID CSVs under a subdirectory inside the configured output directory (e.g., /unknown/).

Behavior details

• In can_reader_loop(), after reading a CAN frame and attempting to match/decode signals:
  • Track whether at least one signal decode+append succeeded for that frame.
  • If no successful decode+append occurred, treat the frame as unknown and append it to the unknown CAN ID CSV for that frame’s ID.
• Unknown frames should be logged regardless of whether signal_table_lookup() returned NULL or returned only placeholder/unusable entries.
• Preserve existing placeholder logic (CAN ID "FFF" in format.json should never be matched/decoded).

Unknown CSV format

• One file per unknown CAN ID, named in a stable, sortable way (recommend always 8 hex digits): unknown_0x%08X.csv.
• File location: /unknown/unknown_0x%08X.csv
• CSV header should be written once when the file is first created.
• Columns (recommended for later forensic analysis):
  • timestamp_ns
  • can_id_hex
  • id_type (SFF for standard 11-bit, EFF for extended 29-bit)
  • dlc
  • data_hex (concatenated hex, dlc bytes)
  • data_hex_spaced (space-separated hex bytes)

Implementation guidance

• Extend writer.c / writer.h with a new API to append unknown frames, or implement a small parallel file-cache for unknown frames similar to existing per-signal FILE* caching.
• Ensure /unknown is created on startup (writer_init is a good place), or lazily create it on first unknown write.
• Ensure file handles are flushed/closed on shutdown (writer_close). If needed, extend writer_t to manage unknown-frame FILE*s.
• Keep performance reasonable (don’t open/close on every frame).

Testing

• Add/update a reproducible end-to-end test using vcan as documented in README.
• The test should demonstrate:
  1. A known ID (e.g. 0x101 in README) produces expected per-signal CSVs.
  2. An unknown ID (choose an ID not in format.json, e.g. 0x123) produces /unknown/unknown_0x00000123.csv and appends a row.
• If the repo already has tests, integrate accordingly; otherwise add a simple shell script under e.g. scripts/ or a documented procedure.

Deliverables

• Code changes implementing unknown CAN ID logging.
• Any necessary documentation updates (README or a new test script) describing how to run the end-to-end smoke test and where the unknown logs appear.

The following is the prior conversation context from the user's chat exploration (may be truncated):

User: what happens if a can id is not found in the json sheet? can it still be saved into a csv?

Assistant: If a CAN frame comes in with an ID that isn’t present in format.json, then in this repo nothing is decoded and nothing is written to CSV for that frame.

Why:

• On each received CAN frame, the code does a lookup by CAN ID:
  • const sig_node_t *node = signal_table_lookup(table, id);
• It then iterates for (; node; node = node->next) and only writes CSV inside that loop after a successful decode (decoder_extract) via writer_append(...).
• If the CAN ID isn’t found, signal_table_lookup(...) returns NULL, the loop body never runs, and no CSV row is produced for that unknown ID. (See src/can_reader.c around the lookup + loop.)

So: No, with the current implementation it will not “still be saved into a CSV” (because the CSV writer is per-signal and only triggered for known signal definitions).

If you want unknown CAN IDs saved anyway, you’d need to add a new behavior, e.g.:

• a “raw frames” CSV like unknown_frames.csv with timestamp_ns,can_id,dlc,data_hex, written when signal_table_lookup(...) returns NULL, or
• a config flag to enable/disable this raw logging.

If you want, tell me the exact CSV format you want for unknown frames and whether you want one file for all unknown IDs or one file per CAN ID, and I’ll point you to the exact place in can_reader_loop() to implement it.

User: in addition to found signals, one csv per unnown can id as well.

Assistant: Yes—this is doable, but it’s not implemented right now. Currently, if a CAN ID isn’t in format.json, the code simply doesn’t call writer_append(), so no CSV gets created for that frame.

To add what you want (“in addition to found signals, create one CSV per unknown CAN ID too”), the cleanest approach is:

Recommended behavior

• Keep the existing per-signal CSVs exactly as-is for known IDs.
• If signal_table_lookup(table, id) returns NULL (or returns only placeholder entries), treat the frame as “unknown” and append it to:
  • `logs/...

This pull request was created from Copilot chat.