Skip to content

[Common] TPC PID: Cleanup of CCDB accessors #13265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jezwilkinson wants to merge 1 commit into
base: master
Choose a base branch
from
+44 −67
Open

[Common] TPC PID: Cleanup of CCDB accessors #13265

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 19 additions & 29 deletions Common/TableProducer/PID/pidTPC.cxx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright 2019-2020 CERN and copyright holders of ALICE O2.

Check failure on line 1 in Common/TableProducer/PID/pidTPC.cxx

View workflow job for this annotation

GitHub Actions / O2 linter

[name/workflow-file] 

Name of a workflow file must match the name of the main struct in it (without the PWG prefix). (Class implementation files should be in "Core" directories.)
// See https://alice-o2.web.cern.ch/copyright for details of the copyright holders.
// All rights not expressly granted are reserved.
//
Expand Down Expand Up @@ -103,22 +103,22 @@
// TPC PID Response
o2::pid::tpc::Response* response;

// CCDB accessor
Service<o2::ccdb::BasicCCDBManager> ccdb;

// Network correction for TPC PID response
OnnxModel network;
o2::ccdb::CcdbApi ccdbApi;
std::map<std::string, std::string> metadata;
std::map<std::string, std::string> nullmetadata;
std::map<std::string, std::string> headers;
std::vector<int> speciesNetworkFlags = std::vector<int>(9);
std::string networkVersion;

// Input parameters
Service<o2::ccdb::BasicCCDBManager> ccdb;
Configurable<std::string> paramfile{"param-file", "", "Path to the parametrization object, if empty the parametrization is not taken from file"};

Check failure on line 117 in Common/TableProducer/PID/pidTPC.cxx

View workflow job for this annotation

GitHub Actions / O2 linter

[name/configurable] 

Use lowerCamelCase for names of configurables and use the same name for the struct member as for the JSON string. (Declare the type and names on the same line.)
Configurable<std::string> url{"ccdb-url", "http://alice-ccdb.cern.ch", "url of the ccdb repository"};

Check failure on line 118 in Common/TableProducer/PID/pidTPC.cxx

View workflow job for this annotation

GitHub Actions / O2 linter

[name/configurable] 

Use lowerCamelCase for names of configurables and use the same name for the struct member as for the JSON string. (Declare the type and names on the same line.)
Configurable<std::string> ccdbPath{"ccdbPath", "Analysis/PID/TPC/Response", "Path of the TPC parametrization on the CCDB"};
Configurable<std::string> recoPass{"recoPass", "", "Reconstruction pass name for CCDB query (automatically takes latest object for timestamp if blank)"};
Configurable<int64_t> ccdbTimestamp{"ccdb-timestamp", 0, "timestamp of the object used to query in CCDB the detector response. Exceptions: -1 gets the latest object, 0 gets the run dependent timestamp"};

Check failure on line 121 in Common/TableProducer/PID/pidTPC.cxx

View workflow job for this annotation

GitHub Actions / O2 linter

[name/configurable] 

Use lowerCamelCase for names of configurables and use the same name for the struct member as for the JSON string. (Declare the type and names on the same line.)
// Parameters for loading network from a file / downloading the file
Configurable<bool> useNetworkCorrection{"useNetworkCorrection", 0, "(bool) Wether or not to use the network correction for the TPC dE/dx signal"};
Configurable<bool> autofetchNetworks{"autofetchNetworks", 1, "(bool) Automatically fetches networks from CCDB for the correct run number"};
Expand All @@ -128,8 +128,8 @@
Configurable<bool> enableNetworkOptimizations{"enableNetworkOptimizations", 1, "(bool) If the neural network correction is used, this enables GraphOptimizationLevel::ORT_ENABLE_EXTENDED in the ONNX session"};
Configurable<int> networkSetNumThreads{"networkSetNumThreads", 0, "Especially important for running on a SLURM cluster. Sets the number of threads used for execution."};
// Configuration flags to include and exclude particle hypotheses
Configurable<int> pidFullEl{"pid-full-el", -1, {"Produce PID information for the Electron mass hypothesis, overrides the automatic setup: the corresponding table can be set off (0) or on (1)"}};

Check failure on line 131 in Common/TableProducer/PID/pidTPC.cxx

View workflow job for this annotation

GitHub Actions / O2 linter

[name/configurable] 

Use lowerCamelCase for names of configurables and use the same name for the struct member as for the JSON string. (Declare the type and names on the same line.)
Configurable<int> pidFullMu{"pid-full-mu", -1, {"Produce PID information for the Muon mass hypothesis, overrides the automatic setup: the corresponding table can be set off (0) or on (1)"}};

Check failure on line 132 in Common/TableProducer/PID/pidTPC.cxx

View workflow job for this annotation

GitHub Actions / O2 linter

[name/configurable] 

Use lowerCamelCase for names of configurables and use the same name for the struct member as for the JSON string. (Declare the type and names on the same line.)
Configurable<int> pidFullPi{"pid-full-pi", -1, {"Produce PID information for the Pion mass hypothesis, overrides the automatic setup: the corresponding table can be set off (0) or on (1)"}};
Configurable<int> pidFullKa{"pid-full-ka", -1, {"Produce PID information for the Kaon mass hypothesis, overrides the automatic setup: the corresponding table can be set off (0) or on (1)"}};
Configurable<int> pidFullPr{"pid-full-pr", -1, {"Produce PID information for the Proton mass hypothesis, overrides the automatic setup: the corresponding table can be set off (0) or on (1)"}};
Expand Down Expand Up @@ -242,21 +242,18 @@
ccdb->setCaching(true);
ccdb->setLocalObjectValidityChecking();
ccdb->setCreatedNotAfter(std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch()).count());
ccdbApi.init(url);
if (time != 0) {
LOGP(info, "Initialising TPC PID response for fixed timestamp {} and reco pass {}:", time, recoPass.value);
ccdb->setTimestamp(time);
response = ccdb->getSpecific<o2::pid::tpc::Response>(path, time, metadata);
headers = ccdbApi.retrieveHeaders(path, metadata, time);
response = ccdb->getSpecific<o2::pid::tpc::Response>(path, time, metadata, &headers);
if (!response) {
LOGF(warning, "Unable to find TPC parametrisation for specified pass name - falling back to latest object");
response = ccdb->getForTimeStamp<o2::pid::tpc::Response>(path, time);
headers = ccdbApi.retrieveHeaders(path, metadata, time);
networkVersion = headers["NN-Version"];
response = ccdb->getForTimeStamp<o2::pid::tpc::Response>(path, time, &headers);
if (!response) {
LOGF(fatal, "Unable to find any TPC object corresponding to timestamp {}!", time);
}
}
networkVersion = headers["NN-Version"];
LOG(info) << "Successfully retrieved TPC PID object from CCDB for timestamp " << time << ", period " << headers["LPMProductionTag"] << ", recoPass " << headers["RecoPassName"];
metadata["RecoPassName"] = headers["RecoPassName"]; // Force pass number for NN request to match retrieved BB
response->PrintAll();
Expand All @@ -274,8 +271,7 @@
if (ccdbTimestamp > 0) {
/// Fetching network for specific timestamp
LOG(info) << "Fetching network for timestamp: " << ccdbTimestamp.value;
bool retrieveSuccess = ccdbApi.retrieveBlob(networkPathCCDB.value, ".", metadata, ccdbTimestamp.value, false, networkPathLocally.value);
headers = ccdbApi.retrieveHeaders(networkPathCCDB.value, metadata, ccdbTimestamp.value);
bool retrieveSuccess = ccdb->getCCDBAccessor().retrieveBlob(networkPathCCDB.value, ".", metadata, ccdbTimestamp.value, false, networkPathLocally.value, "", "", &headers);
networkVersion = headers["NN-Version"];
if (retrieveSuccess) {
network.initModel(networkPathLocally.value, enableNetworkOptimizations.value, networkSetNumThreads.value, strtoul(headers["Valid-From"].c_str(), NULL, 0), strtoul(headers["Valid-Until"].c_str(), NULL, 0));
Expand Down Expand Up @@ -318,32 +314,29 @@
} else {
LOGP(info, "Retrieving TPC Response for timestamp {} and recoPass {}:", bc.timestamp(), recoPass.value);
}
response = ccdb->getSpecific<o2::pid::tpc::Response>(ccdbPath.value, bc.timestamp(), metadata);
headers = ccdbApi.retrieveHeaders(ccdbPath.value, metadata, bc.timestamp());
networkVersion = headers["NN-Version"];
response = ccdb->getSpecific<o2::pid::tpc::Response>(ccdbPath.value, bc.timestamp(), metadata, &headers);
if (!response) {
LOGP(warning, "!! Could not find a valid TPC response object for specific pass name {}! Falling back to latest uploaded object.", metadata["RecoPassName"]);
headers = ccdbApi.retrieveHeaders(ccdbPath.value, nullmetadata, bc.timestamp());
response = ccdb->getForTimeStamp<o2::pid::tpc::Response>(ccdbPath.value, bc.timestamp());
response = ccdb->getForTimeStamp<o2::pid::tpc::Response>(ccdbPath.value, bc.timestamp(), &headers);
if (!response) {
LOGP(fatal, "Could not find ANY TPC response object for the timestamp {}!", bc.timestamp());
}
}
networkVersion = headers["NN-Version"];
LOG(info) << "Successfully retrieved TPC PID object from CCDB for timestamp " << bc.timestamp() << ", period " << headers["LPMProductionTag"] << ", recoPass " << headers["RecoPassName"];
metadata["RecoPassName"] = headers["RecoPassName"]; // Force pass number for NN request to match retrieved BB
response->PrintAll();
}

if (bc.timestamp() < network.getValidityFrom() || bc.timestamp() > network.getValidityUntil()) { // fetches network only if the runnumbers change
LOG(info) << "Fetching network for timestamp: " << bc.timestamp();
bool retrieveSuccess = ccdbApi.retrieveBlob(networkPathCCDB.value, ".", metadata, bc.timestamp(), false, networkPathLocally.value);
headers = ccdbApi.retrieveHeaders(networkPathCCDB.value, metadata, bc.timestamp());
bool retrieveSuccess = ccdb->getCCDBAccessor().retrieveBlob(networkPathCCDB.value, ".", metadata, bc.timestamp(), false, networkPathLocally.value, "", "", &headers);
networkVersion = headers["NN-Version"];
if (retrieveSuccess) {
network.initModel(networkPathLocally.value, enableNetworkOptimizations.value, networkSetNumThreads.value, strtoul(headers["Valid-From"].c_str(), NULL, 0), strtoul(headers["Valid-Until"].c_str(), NULL, 0));
std::vector<float> dummyInput(network.getNumInputNodes(), 1.);
network.evalModel(dummyInput);
LOGP(info, "Retrieved NN corrections for production tag {}, pass number {}, NN-Version number{}", headers["LPMProductionTag"], headers["RecoPassName"], headers["NN-Version"]);
LOGP(info, "Retrieved NN corrections for production tag {}, pass number {}, NN-Version number {}", headers["LPMProductionTag"], headers["RecoPassName"], headers["NN-Version"]);
} else {
LOG(fatal) << "No valid NN object found matching retrieved Bethe-Bloch parametrisation for pass " << metadata["RecoPassName"] << ". Please ensure that the requested pass has dedicated NN corrections available";
}
Expand Down Expand Up @@ -405,10 +398,10 @@
track_properties[counter_track_props + 3] = o2::track::pid_constants::sMasses[species];
track_properties[counter_track_props + 4] = trk.has_collision() ? collisions.iteratorAt(trk.collisionId()).multTPC() / 11000. : 1.;
track_properties[counter_track_props + 5] = std::sqrt(nNclNormalization / trk.tpcNClsFound());
if (input_dimensions == 7 && networkVersion == "2") {

Check failure on line 401 in Common/TableProducer/PID/pidTPC.cxx

View workflow job for this annotation

GitHub Actions / O2 linter

[magic-number] 

Avoid magic numbers in expressions. Assign the value to a clearly named variable or constant.
track_properties[counter_track_props + 6] = trk.has_collision() ? collisions.iteratorAt(trk.collisionId()).ft0cOccupancyInTimeRange() / 60000. : 1.;
}
if (input_dimensions == 8 && networkVersion == "3") {

Check failure on line 404 in Common/TableProducer/PID/pidTPC.cxx

View workflow job for this annotation

GitHub Actions / O2 linter

[magic-number] 

Avoid magic numbers in expressions. Assign the value to a clearly named variable or constant.
track_properties[counter_track_props + 6] = trk.has_collision() ? collisions.iteratorAt(trk.collisionId()).ft0cOccupancyInTimeRange() / 60000. : 1.;
if (trk.has_collision()) {
auto trk_bc = (collisions.iteratorAt(trk.collisionId())).template bc_as<B>();
Expand Down Expand Up @@ -478,10 +471,10 @@
// For now only the option 2: sigma will be used. The other options are kept if there would be demand later on
if (network.getNumOutputNodes() == 1) { // Expected mean correction; no sigma correction
nSigma = (tpcSignal - network_prediction[count_tracks + tracksForNet_size * pid] * expSignal) / expSigma;
} else if (network.getNumOutputNodes() == 2) { // Symmetric sigma correction

Check failure on line 474 in Common/TableProducer/PID/pidTPC.cxx

View workflow job for this annotation

GitHub Actions / O2 linter

[magic-number] 

Avoid magic numbers in expressions. Assign the value to a clearly named variable or constant.
expSigma = (network_prediction[2 * (count_tracks + tracksForNet_size * pid) + 1] - network_prediction[2 * (count_tracks + tracksForNet_size * pid)]) * expSignal;
nSigma = (tpcSignal / expSignal - network_prediction[2 * (count_tracks + tracksForNet_size * pid)]) / (network_prediction[2 * (count_tracks + tracksForNet_size * pid) + 1] - network_prediction[2 * (count_tracks + tracksForNet_size * pid)]);
} else if (network.getNumOutputNodes() == 3) { // Asymmetric sigma corection

Check failure on line 477 in Common/TableProducer/PID/pidTPC.cxx

View workflow job for this annotation

GitHub Actions / O2 linter

[magic-number] 

Avoid magic numbers in expressions. Assign the value to a clearly named variable or constant.
if (tpcSignal / expSignal >= network_prediction[3 * (count_tracks + tracksForNet_size * pid)]) {
expSigma = (network_prediction[3 * (count_tracks + tracksForNet_size * pid) + 1] - network_prediction[3 * (count_tracks + tracksForNet_size * pid)]) * expSignal;
nSigma = (tpcSignal / expSignal - network_prediction[3 * (count_tracks + tracksForNet_size * pid)]) / (network_prediction[3 * (count_tracks + tracksForNet_size * pid) + 1] - network_prediction[3 * (count_tracks + tracksForNet_size * pid)]);
Expand Down Expand Up @@ -553,12 +546,10 @@
} else {
LOGP(info, "Retrieving TPC Response for timestamp {} and recoPass {}:", bc.timestamp(), recoPass.value);
}
response = ccdb->getSpecific<o2::pid::tpc::Response>(ccdbPath.value, bc.timestamp(), metadata);
headers = ccdbApi.retrieveHeaders(ccdbPath.value, metadata, bc.timestamp());
response = ccdb->getSpecific<o2::pid::tpc::Response>(ccdbPath.value, bc.timestamp(), metadata, &headers);
if (!response) {
LOGP(warning, "!! Could not find a valid TPC response object for specific pass name {}! Falling back to latest uploaded object.", metadata["RecoPassName"]);
response = ccdb->getForTimeStamp<o2::pid::tpc::Response>(ccdbPath.value, bc.timestamp());
headers = ccdbApi.retrieveHeaders(ccdbPath.value, nullmetadata, bc.timestamp());
response = ccdb->getForTimeStamp<o2::pid::tpc::Response>(ccdbPath.value, bc.timestamp(), &headers);
if (!response) {
LOGP(fatal, "Could not find ANY TPC response object for the timestamp {}!", bc.timestamp());
}
Expand Down Expand Up @@ -651,12 +642,11 @@
} else {
LOGP(info, "Retrieving TPC Response for timestamp {} and recoPass {}:", bc.timestamp(), recoPass.value);
}
response = ccdb->getSpecific<o2::pid::tpc::Response>(ccdbPath.value, bc.timestamp(), metadata);
headers = ccdbApi.retrieveHeaders(ccdbPath.value, metadata, bc.timestamp());
response = ccdb->getSpecific<o2::pid::tpc::Response>(ccdbPath.value, bc.timestamp(), metadata, &headers);

if (!response) {
LOGP(warning, "!! Could not find a valid TPC response object for specific pass name {}! Falling back to latest uploaded object.", metadata["RecoPassName"]);
response = ccdb->getForTimeStamp<o2::pid::tpc::Response>(ccdbPath.value, bc.timestamp());
headers = ccdbApi.retrieveHeaders(ccdbPath.value, nullmetadata, bc.timestamp());
response = ccdb->getForTimeStamp<o2::pid::tpc::Response>(ccdbPath.value, bc.timestamp(), &headers);
if (!response) {
LOGP(fatal, "Could not find ANY TPC response object for the timestamp {}!", bc.timestamp());
}
Expand Down Expand Up @@ -737,10 +727,10 @@
} else {
LOGP(info, "Retrieving TPC Response for timestamp {} and recoPass {}:", bc.timestamp(), recoPass.value);
}
response = ccdb->getSpecific<o2::pid::tpc::Response>(ccdbPath.value, bc.timestamp(), metadata);
response = ccdb->getSpecific<o2::pid::tpc::Response>(ccdbPath.value, bc.timestamp(), metadata, &headers);
if (!response) {
LOGP(warning, "!! Could not find a valid TPC response object for specific pass name {}! Falling back to latest uploaded object.", metadata["RecoPassName"]);
response = ccdb->getForTimeStamp<o2::pid::tpc::Response>(ccdbPath.value, bc.timestamp());
response = ccdb->getForTimeStamp<o2::pid::tpc::Response>(ccdbPath.value, bc.timestamp(), &headers);
if (!response) {
LOGP(fatal, "Could not find ANY TPC response object for the timestamp {}!", bc.timestamp());
}
Expand Down
10 changes: 4 additions & 6 deletions Common/TableProducer/PID/pidTPCService.cxx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,6 @@ struct pidTpcService {
// CCDB boilerplate declarations
o2::framework::Configurable<std::string> ccdburl{"ccdburl", "http://alice-ccdb.cern.ch", "url of the ccdb repository"};
Service<o2::ccdb::BasicCCDBManager> ccdb;
o2::ccdb::CcdbApi ccdbApi;

o2::aod::pid::pidTPCProducts products;
o2::aod::pid::pidTPCConfigurables pidTPCopts;
Expand All @@ -61,25 +60,24 @@ struct pidTpcService {
ccdb->setCaching(true);
ccdb->setLocalObjectValidityChecking();
ccdb->setCreatedNotAfter(std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch()).count());
ccdbApi.init(ccdburl.value);

// task-specific
pidTPC.init(ccdb, ccdbApi, initContext, pidTPCopts, metadataInfo);
pidTPC.init(ccdb, initContext, pidTPCopts, metadataInfo);
}

void processTracksIU(soa::Join<aod::Collisions, aod::EvSels> const& collisions, soa::Join<aod::TracksIU, aod::TracksCovIU, aod::TracksExtra> const& tracks, aod::BCsWithTimestamps const& bcs)
{
pidTPC.process(ccdb, ccdbApi, bcs, collisions, tracks, static_cast<TObject*>(nullptr), products);
pidTPC.process(ccdb, bcs, collisions, tracks, static_cast<TObject*>(nullptr), products);
}

void processTracksMCIU(soa::Join<aod::Collisions, aod::EvSels> const& collisions, soa::Join<aod::TracksIU, aod::TracksCovIU, aod::TracksExtra, aod::McTrackLabels> const& tracks, aod::BCsWithTimestamps const& bcs, aod::McParticles const&)
{
pidTPC.process(ccdb, ccdbApi, bcs, collisions, tracks, static_cast<TObject*>(nullptr), products);
pidTPC.process(ccdb, bcs, collisions, tracks, static_cast<TObject*>(nullptr), products);
}

void processTracksIUWithTracksQA(soa::Join<aod::Collisions, aod::EvSels> const& collisions, soa::Join<aod::TracksIU, aod::TracksExtra> const& tracks, aod::BCsWithTimestamps const& bcs, aod::TracksQAVersion const& tracksQA)
{
pidTPC.process(ccdb, ccdbApi, bcs, collisions, tracks, tracksQA, products);
pidTPC.process(ccdb, bcs, collisions, tracks, tracksQA, products);
}

PROCESS_SWITCH(pidTpcService, processTracksIU, "Process TracksIU (Run 3)", true);
Expand Down
8 changes: 3 additions & 5 deletions Common/TableProducer/PID/pidTPCServiceRun2.cxx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,6 @@ struct pidTpcServiceRun2 {
// CCDB boilerplate declarations
o2::framework::Configurable<std::string> ccdburl{"ccdburl", "http://alice-ccdb.cern.ch", "url of the ccdb repository"};
Service<o2::ccdb::BasicCCDBManager> ccdb;
o2::ccdb::CcdbApi ccdbApi;

o2::aod::pid::pidTPCProducts products;
o2::aod::pid::pidTPCConfigurables pidTPCopts;
Expand All @@ -61,20 +60,19 @@ struct pidTpcServiceRun2 {
ccdb->setCaching(true);
ccdb->setLocalObjectValidityChecking();
ccdb->setCreatedNotAfter(std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch()).count());
ccdbApi.init(ccdburl.value);

// task-specific
pidTPC.init(ccdb, ccdbApi, initContext, pidTPCopts, metadataInfo);
pidTPC.init(ccdb, initContext, pidTPCopts, metadataInfo);
}

void processTracks(soa::Join<aod::Collisions, aod::EvSels> const& collisions, soa::Join<aod::Tracks, aod::TracksExtra> const& tracks, aod::BCsWithTimestamps const& bcs)
{
pidTPC.process(ccdb, ccdbApi, bcs, collisions, tracks, static_cast<TObject*>(nullptr), products);
pidTPC.process(ccdb, bcs, collisions, tracks, static_cast<TObject*>(nullptr), products);
}

void processTracksMC(soa::Join<aod::Collisions, aod::EvSels> const& collisions, soa::Join<aod::Tracks, aod::TracksExtra, aod::McTrackLabels> const& tracks, aod::BCsWithTimestamps const& bcs, aod::McParticles const&)
{
pidTPC.process(ccdb, ccdbApi, bcs, collisions, tracks, static_cast<TObject*>(nullptr), products);
pidTPC.process(ccdb, bcs, collisions, tracks, static_cast<TObject*>(nullptr), products);
}

PROCESS_SWITCH(pidTpcServiceRun2, processTracks, "Process Tracks", true);
Expand Down
Loading
Loading