Implementation of future test data

[enocera]

The future test data sets, listed on the wiki, are assigned as follows:

dataset	c.m.e. (GeV)	L [fb-1]	arXiv	person in charge
W-boson production differential cross sections	13	140	2502.21088	EC + JK
W+jet production differential cross sections	13	140	2412.11644	KL + AP (on hold)
Z+charm production differential cross sections	13	140	2403.15093	TG + KL
W-boson production differential cross sections (low pile-up)	5.02/13	0.255/0.338	2502.09403	JK + AP (on hold)
W pT cross sections	5.02	0.1	2509.18817	KL+AP

As suggested by @juanrojochacon two new PhD students (+ Ella) are assigned to each data set. The idea is that they independently implement the data set, and then they converge on a common interpretation/implementation. This should be useful for them to be acquainted with the NNPDF commondata framework. @vschutze-alt : I have not assigned you any specific data set because you no longer qualifies as "new" PhD student, but you're more than welcome to help implement any of the data sets, if you'd like to do so. @ecole41 I have assigned only one data set to you. If you want more, just ask. The last data set in the table does not have a Hepdata entry yet.

[juanrojochacon]

Thanks @enocera this is most useful. Sounds like a great plan to me. We should also agree on a timescale to avoid that these things take too long - maybe we can say that by 1 Oct or so there is a first version of the common data implementation that can be cross-checked among the two assigned people?

For Z+charm, maybe @vschutze-alt can also help to speed up things?

Once commondata is implemented, the next step will be to generate NNLO grids, and also here I would adopt the principle that two people do an independent production and then they cross-check each other, in the long term this will save a ton of work.

I understand that progress in future test datasets will be discussed at the code meetings?

[juanrojochacon]

@jekoorn @kamillaurent let me know if anything is unclear, happy to discuss with you about the implementation process in the next days

[juanrojochacon]

Concerning the ATLAS 5 TeV and 13 TeV low-PU cross-sections: despite what Eram said in morimondo, indeed it is still not in in HepData. I will write to him how to see if he can speed up things.

[enocera]

Thanks @enocera this is most useful. Sounds like a great plan to me. We should also agree on a timescale to avoid that these things take too long - maybe we can say that by 1 Oct or so there is a first version of the common data implementation that can be cross-checked among the two assigned people?

Progress will be monitored during the weekly code meetings.

Once commondata is implemented, the next step will be to generate NNLO grids, and also here I would adopt the principle that two people do an independent production and then they cross-check each other, in the long term this will save a ton of work.

I was thinking the same - perhaps two people are not needed to take independent runs of the Monte Carlo, but it may be good to have two people independently implement the runcard which is then passed to a Monte Carlo. Note that in principle this is the mechanism of a PR review.

I understand that progress in future test datasets will be discussed at the code meetings?

Yes

[juanrojochacon]

Well, I think it is still good that two people can carry out independent runs, maybe at low stats, and check that their results agree within MC uncertainties. Then yes, the final high-stats runs do not need to be repeated

[juanrojochacon]

but yes, one step at a time. I already emailed Eram about the missing HepData link, hopefully we will react fast

[kamillaurent]

Hi, thank you for the assignment @enocera. I still don't have access to the wiki though

[juanrojochacon]

Maybe local Edinburgh people like @andrpie or @achiefa can help here?

[achiefa]

I don't have admin access to the wiki unfortunately. I can ask Luigi in person when I see him, which will be next week. Though maybe it would be better to ask him directly tomorrow at the meeting.

[enocera]

Anyways, access to the wiki is not needed to complete this assignment.

[andrpie]

Hi, we've just checked that the data for 2412.11644 is not available online either. Could I work on something else in the meantime? Maybe 2502.21088 if @jekoorn or @ecole41 haven't started yet?

[juanrojochacon]

It is not on HepData @andrpie ? LEt me send an email to the ATLAS conveners - not nice that they don't post their precious datasets on hepdata. If the info maybe available in the paper or somewhere else?

[andrpie]

No, it's not on HepData. I've also checked everywhere else I could think of, to no avail.

[juanrojochacon]

@andrpie I email Eram Rizvi, he promised to get back to us asap.

I discussed a bit yesterday with @kamillaurent and @jekoorn about a couple interesting points concerning the implementation of future test datasets:

• If for the same dataset we have 1d, 2d, or 3d differential measurements, we need three different implementations (same dataset, so the different distributions are mutually exclusive in the fit)
• If a paper presents measurements of different physical processes and their intercorrelation is not available (such as Z+c and Z+b in the ATLAS paper), we implement them as two different datasets

So in a given paper we may have different datasets, one should avoid to think that there is a one to one correspondence between papers and datasets

[juanrojochacon]

Hi @enocera we can discuss tomorrow, but given that two of the ATLAS future test datasets that we planned to implement are not available (and experimentalists sometimes take their time) maybe we consider some other dataset for this exercise, which may be even already available in hepdata? For example from CMS dataset, to make it a bit less ATLAS-centric?

For example, we could implement this very nice measurement of W pT by LHCb in the forward region https://inspirehep.net/literature/2972386 for which the results are provided in a table of the paper (so we don't need HepData). To me this seems a very nice addition to the future tests datasets, and something that deserves to be looked at irrespective of what happens with the ATLAS datasets for which we are waiting for Eram.

What do you think?

[enocera]

@andrpie I email Eram Rizvi, he promised to get back to us asap.

I discussed a bit yesterday with @kamillaurent and @jekoorn about a couple interesting points concerning the implementation of future test datasets:

* If for the same dataset we have 1d, 2d, or 3d differential measurements, we need three different implementations (same dataset, so the different distributions are mutually exclusive in the fit)

* If a paper presents measurements of different physical processes and their intercorrelation is not available (such as Z+c and Z+b in the ATLAS paper), we implement them as two different datasets

So in a given paper we may have different datasets, one should avoid to think that there is a one to one correspondence between papers and datasets

@juanrojochacon Can you please clarify in what your proposal differs from standard NNPDF practice, that I explained in the tutorial? @kamillaurent and @jekoorn are welcome to express their doubts in the PRs related to data set implementation.

[enocera]

Hi @enocera we can discuss tomorrow, but given that two of the ATLAS future test datasets that we planned to implement are not available (and experimentalists sometimes take their time) maybe we consider some other dataset for this exercise, which may be even already available in hepdata? For example from CMS dataset, to make it a bit less ATLAS-centric?

For example, we could implement this very nice measurement of W pT by LHCb in the forward region https://inspirehep.net/literature/2972386 for which the results are provided in a table of the paper (so we don't need HepData). To me this seems a very nice addition to the future tests datasets, and something that deserves to be looked at irrespective of what happens with the ATLAS datasets for which we are waiting for Eram.

What do you think?

Thanks @juanrojochacon this is a nice suggestion. Of course this measurement is so recent that it was not in any of our lists. Remember that I made the list of future data based on what the experimentalists suggested us several months ago. As I said in Morimondo, the beauty of the future test data list is that it's flexible, without affecting the NNPDF4.1 release. So, if you think that the LHCb measurement is interesting, please go with that one. We can dsicuss at the PC and if there's an agreement, I can include it in the list.

[juanrojochacon]

Hi @enocera all is of course aligned with the general policy, we just had a chat about clarifying some points concerning the future tests datasets that they are looking at. And indeed, should they have further questions, they should post them here

[juanrojochacon]

and yes, we will discuss but I think we really need to look at the LHCb dataset. Maybe @jekoorn and @andrpie can look at it together. Let's take stock at the phone conf and make a plan

[juanrojochacon]

I email the LHCb people. In principle all the info we need is available from the paper, but if there is repo with text files then so much the better

[juanrojochacon]

Hi @jekoorn @andrpie the LHCb data is available in HepData

https://www.hepdata.net/record/ins2972386?version=1

as agreed, you can focus on the implementation of this dataset now, and then in the meantime the missing ATLAS data could become available.

Once you are ready, the idea is that one of you opens a pull request with the implementation which is cross-checked by the other.

[juanrojochacon]

I will email CMS in case they have some other "future test dataset" for us to use in this exercise.

[jekoorn]

Hi, I noticed during my implementation that the ATLAS people have refrained from adding the uncertainty on the luminosity of the beam (this 0.83% value) to the uncertainties on HEPData. However, they do note it in the paper as a source of uncertainty. Thsi implies that I will have to manually compute it per bin and add it to the files, correct?

[juanrojochacon]

not really: it a common, fully correlated error which is the same in each bin. So yes, you have to add it manually, but you don't need to "compute it" (unless I misunderstand your question)

[jekoorn]

Ok, that answers my doubt. I will add it

[enocera]

Dear @jekoorn the luminosity uncertainty is given as a percentage, whereas the commondata implementation is such that we specify absolute uncertainties. So, yes, you have to "compute" it for each bin, where the computation amounts to multiplying the percentage luminosity uncertainty by the bin central value, bin by bin. This is then a multiplicative uncertainty, correlated across all ATLAS measurements based on the same luminosity. In this specific case, the run in full Run II, therefore the label to be attached to the luminosity uncertainty is ATLASLUMIRUNII.

[kamillaurent]

Hi, I have finished my first draft of data implementation regarding the article with arXiv number 2403.15093. I express here my major doubts about the paper and things that I might be getting wrong:

• This paper contains a total of 11 different datasets, including differential and inclusive cross section of three processes (where we observe in the final state Z + 1 b-jet, Z + 2 b-jets or Z + 1 c-jet). I don't know if I have to implement every one of these datasets, I have now created a folder for each differential cross section measurement (for a total of 8 folders). I've tried to follow the naming convention, so the folder for the cross section differential in Z transverse momentum in events with Z + 1 b-jet is called ATLAS_Z1B_13TEV_PTZ , and so on

• Each data in HEPData has around 170 associated uncertainties. I am now treating them as follows: "stat --> UNCORR, ADD"; "sys --> CORR, ADD" "Lumi --> MULT, CORR". There are some uncertainties which are labeled as sys,EL_EFF_ID_TOTAL_1NPCOR_PLUS_UNCOR, I am now treating this as "CORR, ADD"

• In the metadata.yaml I am not sure on how to fill the nnpdf31_process: entry and the theory part.

Thanks to anyone willing to answer.

[jekoorn]

And what is the procedure for putting one's implementation up for review? Which branch does one commit the changes into? And do I put also the future test data in the nnpdf_data/commondata/ directory? Thanks

[enocera]

Hi, I have finished my first draft of data implementation regarding the article with arXiv number 2403.15093. I express here my major doubts about the paper and things that I might be getting wrong:

* This paper contains a total of 11 different datasets, including differential and inclusive cross section of three processes (where we observe in the final state Z + 1 b-jet, Z + 2 b-jets or Z + 1 c-jet). I don't know if I have to implement every one of these datasets, I have now created a folder for each differential cross section measurement (for a total of 8 folders). I've tried to follow the naming convention, so the folder for the cross section differential in Z transverse momentum in events with Z + 1 b-jet is called `ATLAS_Z1B_13TEV_PTZ` , and so on

My suggestion here, to avoid a proliferation of folders, is to define the various distributions as different observables of the same data set. So You'll have only one folder, called, say Z+flavoured jets (or similar). Then, in the metadata, you can define the 8 observables. Before proceeding you may want to give me a couple of days to look into the paper and check whether we want to implement all of the observables.

* Each data in HEPData has around 170 associated uncertainties. I am now treating them as follows: "stat --> UNCORR, ADD"; "sys --> CORR, ADD" "Lumi --> MULT, CORR".  There are some uncertainties which are labeled as `sys,EL_EFF_ID_TOTAL_1NPCOR_PLUS_UNCOR`, I am now treating this as "CORR, ADD".

The statistical uncertainty, if an unfolding covariance matrix is not provided, is surely ADD, UNCORR. The luminosity uncertainty is MULT, however you should be careful and correlate it with all the other measurements performed by the same experiment within the same run. In this specific case, the measurement is from ATLAS and makes use of the full Run II luminosity, therefore the correlation label to assign to the luminosity uncertainty is ATLASLUMI15. That same label will be assigned to the luminosity uncertainty of all of the ATLAS measurements based on the full Run II. As for the other uncertainties, I suggest that these be checked against the proposal of the other person in charge of implementing the data set.

* In the `metadata.yaml` I am not sure on how to fill the `nnpdf31_process:` entry and the `theory` part.

That's a good question. I would label it as DY_NC for the time being. You can skip the theory part as of now.

[enocera]

And what is the procedure for putting one's implementation up for review? Which branch does one commit the changes into? And do I put also the future test data in the nnpdf_data/commondata/ directory? Thanks

Please create a branch with a different name. I suggest that you use the same directory structure nnpdf_data/commondata. Future test data may become regular data sets in the future.