Chek the ATLAS Z0 8 TeV low mass data set

[achiefa]

This PR addresses #2267.

[achiefa]

Dear @enocera,

As mentioned in issue #2267, I've implemented the "light" variant of the dataset. In this variant, statistical and systematic uncertainties are taken from Table 6 in 1710.05167, on top of which I included the luminosity uncertainty (1.9%). However, I took the liberty to play with the luminosity uncertainty and I combined it with different variants, which are listed below:

1. HepData v3 (ATLASLUMI12): Here I took version 3 of the HepData table and I correlated the luminosity uncertainty using the key ATLASLUMI12. This corresponds to the new default implementation.
2. HepData v3 (CORR LUMI): As (1), but now the luminosity is correlated within the same experiment (key CORR).
3. HepData v3 (UNCORR LUMI): As (1), but now the luminosity is not correlated at all (key UNCORR).
4. HepData v1 (ATLASLUMI12): As (1), but with version 1 in HepData
5. HepData v1 (CORR LUMI): As (2), but with version 1 in HepData
6. HepData v1 (UNCORR LUMI): As (3), but with version 1 in HepData
7. Syst. light (ATLASLUMI12): As (1), but using the light version of systematic uncertainties.
8. Syst. light (CORR LUMI): As (2), but using the light version of systematic uncertainties.
9. Syst. light (UNCORR LUMI): As (3), but using the light version of systematic uncertainties.

For each of these variants, I computed the $\chi^2$ to ATLAS_Z0_8TEV_LOWMASS using the same pdf set (250122-jth-01-data-check, the one used by RS in his report). You can find the report here.

I think the message is rather clear -- the $\chi^2$ improves when the luminosity uncertainty is not correlated. This is true for version 3 on HepData and the light version (3 and 9 in the list above). On the other hand, the $\chi^2$ for version 1 is always high (~50) regardless of the type of correlation assigned to the luminosity.

The list above should provide a comprehensive set of combinations useful for this investigation. Please, let me know if something is missing. Also, I've pushed the uncertainty data files that I used to produce the variants, so that you can have a look if something does not convince you.

P.S. In the original re-implementation, the luminosity uncertainty was set to 1.8%, although the paper claims 1.9%. Apart from version 1, which already includes (God knows what) luminosity uncertainty in the HepData table, all the other variants use 1.9%.

[scarlehoff]

Some results from today's code meeting, using directly the NNLO grids (which are slightly different) to compute the chi2 and NNPDF40_nnlo_as_01180 as PDF.

legacy: 0.6911 default: 8.743 default with CT18Z: 5.649

[achiefa]

Dear @enocera $ @scarlehoff,

As agreed in the last code meeting, I computed the values of the $\chi^2$ for each bin in the invariant mass of the lepton pair. The dataset comes with seven bins, although the last two are excluded by the cuts in the NNPDF4.0. For each bin, I computed the $\chi^2$ using all the dataset variants that I discussed in the previous comment, and you can find the reports below (bins are labelled as $[m_{\ell\ell, min}, m_{\ell\ell, max}]$ GeV):

1. $[46, 66]$: link
2. $[66, 80]$: link
3. $[80, 91]$: link
4. $[91, 102]$: link
5. $[102, 116]$: link
6. $[116, 150]$: link
7. $[150, 200]$: link

First of all, version 1 of HepData always results in an odd $\chi^2$ regardless of the bin and of the correlation of the luminosity uncertainty. Therefore, I think we can rule it out once and for all.

Second, the $\chi^2$ for the new implementation is good for bins (1), (2), and (5), which incidentally are those that do not include the $Z$-peak. Furthermore, for bins (2) and (5) the $\chi^2$ of the new implementation is better than the legacy value upon inclusion of the correlation of the luminosity.

Finally, the origin of the crazily high $\chi^2$ that we observed might origin from the bins that include the peak at $m_Z$, namely (3) and (4). Indeed, there we can see the problematic values of the $\chi^2$ when we account for the correlation of the luminosity. Note that, for these bins, the "light" version does not help reduce the $\chi^2$ either.

[scarlehoff]

Thank you very much for this @achiefa Is this the same dataset that was problematic in the pheno paper? I thought we had checked that the old version was ok even for the peak or is this just another dataset which happens to also have problems in the peak?

edit: silly me, the old version was always legacy so luminosity was always uncorrelated

[achiefa]

edit: silly me, the old version was always legacy so luminosity was always uncorrelated

Indeed, I was just about to say that.

edit: But maybe that can explain why we could not describe well the new ATLAS analysis at $Z$-peak. That's just me thinking out loudly, but maybe the new ATLAS analysis, which is at the $Z$-peak, fixes the problematic correlation of the luminosity. If that is true, then the old dataset (ATLAS_Z0_8_TEV_LOWMASS @ $Z$-peak) and the new one are simply incompatible.

[enocera]

edit: But maybe that can explain why we could not describe well the new ATLAS analysis at Z -peak. That's just me thinking out loudly, but maybe the new ATLAS analysis, which is at the Z -peak, fixes the problematic correlation of the luminosity. If that is true, then the old dataset (ATLAS_Z0_8_TEV_LOWMASS @ Z -peak) and the new one are simply incompatible.

Very well thought - this is the explanation I'm leaning towards.

[scarlehoff]

The new one is incompatible also with other datasets though, since we were not able to fit it by itself without mhou...

[enocera]

@achiefa Question: all these chi2 are w/o MHOUs, right? So that we can compare with the old NNPDF4.0 fit?

[achiefa]

Yes, all those chi2's are w/o MHOUs.

[enocera]

Dear @achiefa I looked into the MSHT paper, see in particular Fig. 7 of https://arxiv.org/pdf/2012.04684. They have cuts on the ATLAS 8 TeV 2D low mass data, in particular they exclude, for each invariant mass bin, the two most forward rapidity bins. They also remove the lowest invariant mass bin altogether. Could you please try to recompute the chi2 with these cuts? Thanks.

[achiefa]

Hi @enocera,

I will compute the chi2. On top of the cuts that you mentioned, should I relax the cuts that remove the two last bins in the invariant mass?

[enocera]

In our framework, we want to always remove the last two invariant mass bins, because these are incorporated in the high-mass data set. So I would NOT relax the cuts and continue to exclude the last two invariant mass bins.

[achiefa]

Follow-up

I computed the $\chi^2 as requested by @enocera, namely applying the same cuts as MSHT. You can find the report here. As far as I can understand, I don't see any improvement in the $\chi^2$ - actually, the value worsens in any dataset variant.

I also computed the $\chi^2$ for two other configurations:

• computing the $\chi^2$ for all the bins but for those at the Z peak using the same set as in the previous comments (250122-jth-01-data-check). The last two invariant mass bins are still excluded. The report is here.
• computing the $\chi^2$ using 241003-01-rs-mhou which includes MHOUs. The report is here.

I'll compute the $\chi^2$ for other configurations, for instance as done in the pheno paper where the calculation of the $\chi^2$ includes the covmat for MHOUs.

[enocera]

Thanks, Amedeo. So should we conclude that, as soon as we put all the invariant mass bins together (but those on the Z peak), we get a horrible chi2?

[achiefa]

I think it depends on what we are looking at. The report in the first bullet point should be compared with the report I generated for the first comment (here for brevity). Then what I see is that the legacy implementation - which does not correlate the luminosity - worsens, but the $\chi^2$ improves when we correlate the luminosity, that is for the variant implemented by Mark (version 3).

[scarlehoff]

I've been talking a bit with Tom about this problem and trying to compare our data to theirs... statistical uncertainties and data seem to be the same, and the 200+ systematics are impossible to compare well but one thing I noticed is that their luminosity uncertainty is 1.9% while we have 1.8% why is that? Looking at the paper that we reference for this it says 1.9% https://arxiv.org/pdf/1710.05167

[enocera]

This is done on purpose. The luminosity uncertainty is 100% correlated across different ATLAS experiments, therefore, even if the paper says 1.9%, I set it up to 1.8%, which is the value (also mentioned in ATLAS papers) corresponding to that specific 8 TeV run. You can try to have a variant in which this is set to 1.9% and check the difference. I bet that this is not going to solve the problem.

[scarlehoff]

No, it doesn't. It's too small of a change, but this was a difference between the two (and different from the paper we reference in the code to say 1.8, that should probably be changed). I was hoping to find something else unraveling that thread.

[enocera]

Maybe the MSHT crowds have decorrelated some specific uncertainties? Or perhaps they are not correlating (certain) uncertainties across invariant mass bins but only across rapidity bins? That would likely explain the good chi2: indeed, we have a good chi2 for each invariant mass bin separately - and we would have a good global chi2 if uncertainties in the various invariant mass bins were not 100% correlated.

[enocera]

Maybe the MSHT crowds have decorrelated some specific uncertainties? Or perhaps they are not correlating (certain) uncertainties across invariant mass bins but only across rapidity bins? That would likely explain the good chi2: indeed, we have a good chi2 for each invariant mass bin separately - and we would have a good global chi2 if uncertainties in the various invariant mass bins were not 100% correlated.

This is a silly observation. we know that problems arise only when we correlate the luminosity uncertainty.

[scarlehoff]

This is a silly observation. we know that problems arise only when we correlate the luminosity uncertainty.

I don't think it is. We resolve the problems when we correlate the luminosity but that one in particular is being implemented exactly in the same way by us and them. But what might well happen is that we are correlating differently some other uncertainties and by uncorrelating the luminosity we are compesating for it. This point about correlating over rapidity / mass might be relevant, I'll ask about it.

[enocera]

@scarlehoff Thanks for checking with Tom.

[scarlehoff]

This point about correlating over rapidity / mass might be relevant, I'll ask about it.

No, in principle they correlate everything just as we do. inspecting by eye* numbers seem also equivalent. But if I try to get the chi2/N value using validphys, using our data and our predictions with their cuts, I get a different value for the chi2 that they do in their paper (I get ~10, they show ~2 in the MSHT20 paper)

meta:
  title: Comparison
  keywords: comparison, 
  author: juacrumar

pdfs:
  - NNPDF40_nnlo_as_01180
  - MSHT20nnlo_as118

theoryid: 40_000_000
use_cuts: "internal"
use_pdferr: false

drop_internal_rules:
  - "ATLAS_Z0_8TEV_LOWMASS_M-Y"

added_filter_rules:
  - dataset: ATLAS_Z0_8TEV_LOWMASS_M-Y
    rule: m_Z2 <= 40000
  - dataset: ATLAS_Z0_8TEV_LOWMASS_M-Y
    rule: m_Z2 >= 4356
  - dataset: ATLAS_Z0_8TEV_LOWMASS_M-Y
    rule: abs_y <= 2.0

dataset_inputs:
  - { dataset: "ATLAS_Z0_8TEV_LOWMASS_M-Y"}

template_text: |
  {@ plot_datasets_pdfs_chi2 @}

actions_:
  - report(main=true)

*there are many numbers, inspection by eye not very reliable.