Options/Skip Test Sources should be default checked
Most likely setting is to not scan test sources, make that the default. Or it should be taken from a project initial value, or from another project window. When opening a project where pmd is not used on, yet, with another project open, it takes the ruleset from the other project but not the Skip Test Sources setting.
I wonder why we have the decision of scanning test sources as a project global option anyway? Wouldn't it make more sense to have such a property per rule set?
E.g., in my projects I do have different rule sets for tests and production code. So in IntelliJ it would be helpful to specify for each new ruleset if those rules should be applied to test sources or production sources (or both).
Similar to CheckStyle: