python
gh-119127: functools.partial placeholders
As I already had implementation I though PR might be helpful for others to see and evaluate.
From all the different extensions of functools.partial I think this one is the best. It is relatively simple and exposes all missing functionality. Other partial extensions that I have seen lack functionality and would not provide complete argument ordering capabilities and/or are too complicated in relation to what they offer.
Implementation can be summarised as follows:
a) Trailing placeholders are not allowed. (Makes things simpler)
b) Throws exception if not all placeholders are filled on call
c) retains optimization benefits of application on other partial instances.
Performance penalty compared to current functools.partial is minimal for extension class. + 20-30 ns for initialisation and <4 ns when called with or without placeholders.
To put it simply, new functionality extends functools.partial so that it has flexibility of lambda / def approach (in terms of argument ordering), but call overhead is 2x smaller.
The way I see it is that this could only be justified if this extension provided completeness and no new functionality is going to be needed anywhere near in the future. I have thought about it and tried various alternatives and I think there is a good chance that this is the case. Personally, I don't think I would ever need anything more from partial class.
Current implementation functions reliably.
Benchmark
There is nothing new here in terms of performance. The performance after this PR will be (almost) the same as the performance of partial until now. Placeholders only provide flexibility for taking advantage of performance benefits where it is important.
So far I have identified 2 such cases:
- More flexible predicate construction for functions in
operatormodule. This allows for new strategies in making performantiteratorrecipes. -
Partializinginput target function. Examples of this are optimizers and similar. I.e. cases where the function will be called over and over within the routine with number of arguments. But the input target function needs partial substitution for positionals and keywords.
Good example of this is scipy.optimize.minimize.
Its signature is: scipy.optimize.minimize(fun, x0, args=(), ...)
Note, it does not have kwds. Why? I don't know. But good reason for it could be:
fun = lambda x: f(x, **kwds)
will need to expand **kwds on every call (even if it is empty), while partial will make the most optimal call. (see benchmarks below). So the minimize function can leave out kwds given there is a good way to source callable with already substituted keywords.
This extension allows pre-substituting both positionals and keywords. This allows optimizer signature to leave out both kwds and args resulting in simpler interface scipy.optimize.minimize(fun, x0, ...) and gaining slightly better performance - function calls are at the center of such problems after all.
Benchmark Results for __call__
Code for Cases
dct = {'a': 1}
kwds = {'c': 1, 'd': 2}
kwds_empty = {}
args1 = (1,)
args3 = (1, 2, 4)
opr_sub = opr.sub
opr_contains = opr.contains
opr_sub_lambda = lambda b: opr_sub(1, b)
opr_sub_partial = ftl.partial(opr_sub, 1)
opr_contains_lambda = lambda b: opr_contains(dct, b)
opr_contains_partial = ftl.partial(opr_contains, dct)
def pos2(a, b):
pass
def pos6(a, b, c, d, e, f):
pass
def pos2kw2(a, b, c=1, d=2):
pass
pos2_lambda = lambda b: pos2(1, b)
pos2_partial = ftl.partial(pos2, 1)
pos6_lambda = lambda b, c, d: pos6(1, 2, 3, b, c, d)
pos6_partial = ftl.partial(pos6, 1, 2, 3)
pos2kw2_kw_lambda = lambda b: pos2kw2(1, b, **kwds)
pos2kw2_kw_partial = ftl.partial(pos2kw2, 1, **kwds)
pos2kw2_kwe_lambda = lambda b: pos2kw2(1, b, **kwds_empty)
pos2kw2_kwe_partial = ftl.partial(pos2kw2, 1, **kwds_empty)
opr_sub_partial_ph = ftl.partial(opr_sub, PH, 1)
opr_contains_partial_ph = ftl.partial(opr_contains, PH, 'a')
pos2_partial_ph = ftl.partial(pos2, PH, 1)
pos6_partial_ph = ftl.partial(pos6, PH, 2, PH, 4, PH, 6)
pos2kw2_kw_partial_ph = ftl.partial(pos2kw2, PH, 1, **kwds)
pos2kw2_kwe_partial_ph = ftl.partial(pos2kw2, PH, 1, **kwds_empty)
# Placeholder versions
from functools import Placeholder as PH
opr_sub_partial_ph = ftl.partial(opr_sub, PH, 1)
opr_contains_partial_ph = ftl.partial(opr_contains, PH, 'a')
pos2_partial_ph = ftl.partial(pos2, PH, 1)
pos6_partial_ph = ftl.partial(pos6, PH, 2, PH, 4, PH, 6)
pos2kw2_kw_partial_ph = ftl.partial(pos2kw2, PH, 1, **kwds)
pos2kw2_kwe_partial_ph = ftl.partial(pos2kw2, PH, 1, **kwds_empty)
CPython Results
C Implementation
----------------
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ 5 repeats, 1,000,000 times ┃
┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Units: ns lambda partial ┃
┃ ┏━━━━━━━━━━━━━━━━━━━┫
┃ opr_sub ┃ 50 ± 4 40 ± 2 ┃
┃ opr_contains ┃ 53 ± 3 43 ± 3 ┃
┃ pos2 ┃ 50 ± 1 64 ± 1 ┃
┃ pos2(*args1) ┃ 69 ± 5 73 ± 5 ┃
┃ pos6 ┃ 58 ± 1 103 ± 5 ┃
┃ pos6(*args3) ┃ 77 ± 3 99 ± 5 ┃
┃ pos2kw2_kw ┃ 240 ± 4 259 ± 7 ┃
┃ pos2kw2_kwe ┃ 134 ± 6 69 ± 3 ┃
┗━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━┛
With Placeholders
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ 5 repeats, 1,000,000 times ┃
┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Units: ns lambda partial Placeholders ┃
┃ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ opr_sub ┃ 50 ± 2 39 ± 1 44 ± 4 ┃
┃ opr_contains ┃ 61 ± 2 44 ± 2 49 ± 2 ┃
┃ pos2 ┃ 54 ± 2 58 ± 3 64 ± 2 ┃
┃ pos2(*args1) ┃ 67 ± 3 72 ± 9 69 ± 3 ┃
┃ pos6 ┃ 63 ± 3 102 ± 3 99 ± 2 ┃
┃ pos6(*args3) ┃ 75 ± 3 101 ± 2 94 ± 4 ┃
┃ pos2kw2_kw ┃ 242 ± 7 259 ± 10 260 ± 7 ┃
┃ pos2kw2_kwe ┃ 131 ± 4 64 ± 1 69 ± 2 ┃
┗━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
Python Implementation
---------------------
Current
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ 5 repeats, 1,000,000 times ┃
┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Units: ns lambda partial ┃
┃ ┏━━━━━━━━━━━━━━━━━━━━━┫
┃ opr_sub ┃ 48 ± 1 373 ± 13 ┃
┃ opr_contains ┃ 51 ± 1 377 ± 12 ┃
┃ pos2 ┃ 51 ± 4 378 ± 5 ┃
┃ pos2(*args1) ┃ 63 ± 5 354 ± 7 ┃
┃ pos6 ┃ 59 ± 1 437 ± 5 ┃
┃ pos6(*args3) ┃ 75 ± 2 410 ± 7 ┃
┃ pos2kw2_kw ┃ 239 ± 4 517 ± 5 ┃
┃ pos2kw2_kwe ┃ 133 ± 3 408 ± 49 ┃
┗━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━┛
With Placeholders
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ 5 repeats, 1,000,000 times ┃
┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Units: ns lambda partial Placeholders ┃
┃ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ opr_sub ┃ 49 ± 1 392 ± 13 547 ± 12 ┃
┃ opr_contains ┃ 54 ± 2 393 ± 9 605 ± 78 ┃
┃ pos2 ┃ 55 ± 9 398 ± 7 544 ± 5 ┃
┃ pos2(*args1) ┃ 66 ± 2 373 ± 5 533 ± 8 ┃
┃ pos6 ┃ 58 ± 5 462 ± 4 652 ± 3 ┃
┃ pos6(*args3) ┃ 74 ± 2 428 ± 11 635 ± 9 ┃
┃ pos2kw2_kw ┃ 240 ± 5 533 ± 4 696 ± 10 ┃
┃ pos2kw2_kwe ┃ 134 ± 2 406 ± 4 555 ± 3 ┃
┗━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
PyPy Results
PyPy
----
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ 5 repeats, 10,000 times ┃
┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Units: ns lambda partial ┃
┃ ┏━━━━━━━━━━━━━━━━━━━━━┫
┃ opr_sub ┃ 122 ± 15 266 ± 70 ┃
┃ opr_contains ┃ 147 ± 7 248 ± 64 ┃
┃ pos2 ┃ 114 ± 17 204 ± 49 ┃
┃ pos2(*args1) ┃ 156 ± 24 202 ± 28 ┃
┃ pos6 ┃ 124 ± 14 268 ± 39 ┃
┃ pos6(*args3) ┃ 147 ± 36 225 ± 21 ┃
┃ pos2kw2_kw ┃ 259 ± 17 436 ± 66 ┃
┃ pos2kw2_kwe ┃ 180 ± 14 243 ± 43 ┃
┗━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━┛
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ 5 repeats, 1,000,000 times ┃
┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Units: ns lambda partial ┃
┃ ┏━━━━━━━━━━━━━━━━━━┫
┃ opr_sub ┃ 1 ± 0 3 ± 1 ┃
┃ opr_contains ┃ 13 ± 0 16 ± 2 ┃
┃ pos2 ┃ 1 ± 0 3 ± 1 ┃
┃ pos2(*args1) ┃ 2 ± 0 2 ± 0 ┃
┃ pos6 ┃ 1 ± 0 2 ± 0 ┃
┃ pos6(*args3) ┃ 2 ± 0 2 ± 0 ┃
┃ pos2kw2_kw ┃ 42 ± 1 72 ± 2 ┃
┃ pos2kw2_kwe ┃ 2 ± 0 2 ± 0 ┃
┗━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━┛
PyPy Placeholder
----------------
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ 5 repeats, 10,000 times ┃
┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Units: ns lambda partial Placeholders ┃
┃ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ opr_sub ┃ 114 ± 5 256 ± 82 719 ± 170 ┃
┃ opr_contains ┃ 142 ± 7 538 ± 536 787 ± 145 ┃
┃ pos2 ┃ 125 ± 19 239 ± 54 679 ± 116 ┃
┃ pos2(*args1) ┃ 130 ± 30 199 ± 17 638 ± 48 ┃
┃ pos6 ┃ 115 ± 16 237 ± 43 785 ± 176 ┃
┃ pos6(*args3) ┃ 138 ± 25 214 ± 14 703 ± 19 ┃
┃ pos2kw2_kw ┃ 260 ± 24 382 ± 67 850 ± 92 ┃
┃ pos2kw2_kwe ┃ 179 ± 28 223 ± 44 661 ± 32 ┃
┗━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ 5 repeats, 1,000,000 times ┃
┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Units: ns lambda partial Placeholders ┃
┃ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ opr_sub ┃ 1 ± 0 3 ± 1 156 ± 4 ┃
┃ opr_contains ┃ 13 ± 0 15 ± 1 173 ± 3 ┃
┃ pos2 ┃ 2 ± 0 3 ± 1 154 ± 7 ┃
┃ pos2(*args1) ┃ 2 ± 0 2 ± 0 148 ± 3 ┃
┃ pos6 ┃ 2 ± 0 3 ± 1 200 ± 2 ┃
┃ pos6(*args3) ┃ 2 ± 0 3 ± 0 217 ± 39 ┃
┃ pos2kw2_kw ┃ 43 ± 1 71 ± 1 240 ± 2 ┃
┃ pos2kw2_kwe ┃ 2 ± 0 2 ± 0 149 ± 2 ┃
┗━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
Setup:
- First 2 columns are identical calls - one using
lambdaotherpartial. - 3rd column is using placeholder to expose 1st argument as opposed to 2nd (or different places for 6-arg case).
CPython:
- There is negligible impact on
__call__. Run times are very close of current and new version with Placeholders. - It can be seen that run times are not impacted by placeholder usage in any significant way.
-
pos2kw2_kwe(emptykwds) is much faster ofpartialcall.pos2kw2_kw(non-emptykwds) is currently slower, however https://github.com/python/cpython/pull/120783 will likely to improve its speed so that it outperforms lambda.
PyPy:
- Usage of
Placeholdersresults in very poor performance. However, this has no material implication aslambdais more performant thanpartialin all cases and is an optimal choice.
Benchmark Results for __new__
INIT='import functools as ftl; g = lambda a, b, c, d, e, f: (a, b, c, d, e, f);'
# CURRENT
$PY_MAIN -m timeit -s $INIT 'ftl.partial(g, 0, 1, 2)' # 160
# PLACEHOLDERS
INIT2="$INIT PH=ftl.Placeholder;"
$PY_MAIN -m timeit -s $INIT2 'ftl.partial(g, 0, 1, 2)' # 170
$PY_MAIN -m timeit -s $INIT2 'ftl.partial(g, 0, 1, 2, 3, 4, 5)' # 190
$PY_MAIN -m timeit -s $INIT2 'ftl.partial(g, PH, 1, PH, 3, PH, 5)' # 200
- There is small performance decrease for initialization without placeholders.
- Initializing it with placeholders is slower for the same number of arguments (excluding placeholders).
- But it is not much slower if placeholders are counted as arguments.
To sum up
This extension:
- allows extracting current performance benefits of
partialto few more important (at least from my POV) cases. - seems to allow for certain simplifications to happen by bringing it more in line with
lambda/defbehaviour. Thus, allowingpartialto be used forpartialmethodapplication which allows for some simplifications in handling these in other parts of the library - i.e.inspect.
It is a flexibility improvement for already existing partial object and should not be seen as an attempt to provide a substitute for more general lambda/def.
partial with (or without) placeholders should be used where appropriate. It could be preferred if:
- It offers required performance benefits
- Arguments need to be hard-bound at the time of partialization so that subsequent changes in closure have no impact. E.g. binding arguments generated by the loop.
If unsure, one should use lambda/def.
- Issue: gh-119127
📚 Documentation preview 📚: https://cpython-previews--119827.org.readthedocs.build/
I personally would advocate for
PLACEHOLDERinstead ofPlaceholderto stress that 1) this is not a global constant, just a module constant, 2) this is not a class, 3) this is named similarly todataclasses.KW_ONLY.
If this was python implementation only, I would agree, but C implementation closely follows None and other global constants. I think it would be more correct to follow that pattern given its nature and in turn emulate it as closely in python implementation as possible.
C implementation:
type(Placeholder)() is Placeholder
# same as None
type(None)() is None
If this renders like that, I think you can ignore my comments on the uppercasing then. Nonetheless, you should definitely document the new sentinel in the functools module, the same way it is done for None (I assume).
If this renders like that, I think you can ignore my comments on the uppercasing then. Nonetheless, you should definitely document the new sentinel in the
functoolsmodule, the same way it is done forNone(I assume).
I will think about it a bit more. UPPERCASE has its benefits of being distinct. E.g. my own sentinels:
AUTO
NULL
EMPTY
It is hard to miss them and they have a solid look of C constants. Also, syntax highlighting will not kick in for CamelCase (at least for a while).
Will let it sit for a while, see what others think. Thank you.
I thought about it and I am leaning to leave it with CamelCase. I think everyone can just assign it to whatever they like, be it _, PH or PLACEHOLDER. And for actual definition probably best to leave it in harmony with similar implementations.
predicate = itl.partial(isinstance, Placeholder, str)
Actually, all the "good" usecases, IMO, boil down to a right partialization instead of a left partialization as I said in https://github.com/python/cpython/issues/119127#issuecomment-2155982139.
predicate = itl.partial(isinstance, Placeholder, str)
Actually, all the "good" usecases, IMO, boil down to a right partialization instead of a left partialization as I said in #119127 (comment).
Would you be kind to provide some support for your statement?
"all" is a strong word. Such statement should come with some measurable comparison between left and right usecases in "goodness" and "quantity" at the very least.
From what I've seen in the discussion (issue and discourse), all the (non-trivial) examples are actually examples of right partialization (be it by yours or Raymond's). Most of the time, the issue with partial is when you cannot use keyword arguments because they are either pos-only or they have non-meaningful names.
I do not have the numbers, nor will I bother to find them, but I'm pretty sure that I can replace 'all' by 'almost all' if you think it's a too strong word. While partialization in some arbitrary place would be useful, e.g., f(x, y, z, t) and you want to bind x=1, y=2, t=3 so that you have something like f_z = partial(f, 1, 2, _, 3), I think it would conceptually be simpler in this case to write a two-line function instead (for clarity and to explain the constants), those cases are (in the examples that are shown) rarer. So, I think that adding rpartial with a simpler logic would just solve most of the examples.
rpartial would mean 1 more Python class and 1 more C class.
It would be at least twice as much code + one more class which needs to be kept in sync between 2 implementations. The simplicity of implementation would be offset by the complexity introduced by amount of code added.
Also, the total flexibility of partial + rpartial would still be less than partial+Placeholder. Placeholder will cover 99% of use-cases, while partial+rpartial maybe would amount to around 90%. So there would still be a portion of unsatisfied users which would result in reoccurring posts from time to time with references to external packages that are able to handle cases that standard library can not.
Readability of Placeholders is also better than rpartial, because when you look at definition:
func = lambda: a, b, c, d
partial(func, _, 1, _, 2)
You can see the order and unfilled places and directly compare with partialized function to work out where goes what.
In contrast with rpartial, where one would need to count the number of arguments from the right and see what is left.
So I don't really see an argument for rpartial from any POV.
I don't think it's hard to maintain since partials are not needed to be affected a lot. While I agree that it's somewhat a code duplication, I would be glad to have just rpartial without bothering about my placeholders. Also, the duplication is quite simple to avoid since you just need to generate code depending on the direction of the arguments being filled so it's not really an issue IMO (right partialization can be regarded as a left partialization of a function with its arguments filled in reverse order).
will cover 99% of use-cases, while partial+rpartial maybe would amount to around 90%. So there would still be a portion of unsatisfied users which would result in reoccurring posts from time to time with references to external packages that are able to handle cases that standard library can
Well... I can ask you also to give me numbers for those 9% additional cases but I'd just say that you cannot satisfy every user. Some could just say "but this does not support well default values!" (rpartial wouldn't either but is it only 1% of those non-covered cases?).
Readability of Placeholders is also better than rpartial, because when you look at definition:
Your definition is not a right partialization so it cannot be compared... I just wanted to highlight that most, if not all examples, actually require a right partialization support rather than a generic one. This is the only thing I wanted to stress.
In contrast with rpartial, where one would need to count the number of arguments from the right and see what is left.
Well... you are assuming that your definition and your partialization are close to each other which is not necessarily the case. The example of divmod where you bind the divisor, of bin_8b where you bind the format or isinstance where you bind the type could all be written without using the placeholder. For instance,
is_int = rpartial(isinstance, int)
is clearer (for me) than
is_int = partial(isinstance, PLACEHOLDER, int)
While I don't reject the idea of a placeholder nor a generic partial (I would be more interested in a singledispatch where I can choose which argument to dispatch over but that's another question), I would prefer not using placeholders if I could.
Now, other languages have kind of a similar interface. For Java, you have a bindTo, where only the first argument are bound. For C++ you have std::bind_front and std::bind_back (equivalent of lpartial and rpartial) but you also support std::bind for arbitrary binding (though you can even reorder the arguments call order). For Javascript, you only have left partialization using bind but you have lodash with partialRight and bind (the first is only for right partialization, the second is like the C++ bind). And for Scala you only have arbitrary binding. So The fact that a right partialization exists in languages is also an indication of some user needs, so we could also have both a partial and rpartial and a generic bind_func like in C++ where you would use placeholders as well, but only if needs arise.
- You fail to digest what I have said
- The type and quality of argumentation you are trying to have here is appropriate on discourse at the initial stages, not when the PR is almost done.
Also, the duplication is quite simple to avoid since you just need to generate code depending on the direction of the arguments being filled so it's not really an issue IMO.
Explain how would you avoid duplication. Please be concrete, not at the contemplation level.
I just wanted to highlight that most, if not all examples, actually require a right partialization support rather than a generic one.
You have not taken into account cases that were not presented here and cases that would spontaneously arise once the utility is at disposal. Finally, why have less flexibility if you can have more?
is clearer (for me) than
partial(isinstance, _, int)
to me is clearer. It does not require 2-directional thinking. Signatures are written from left to right. rpartial introduces a need to think about order from right to left in addition.
All in all, your position is a radical challenge to what has been done so far. Which I wouldn't mind if you either:
- Wrote a PM so we can have a chat without spamming PR so I can address your concerns. You surely understand that I have thought about this much more than you did by now.
- Brought this up in discourse
- Wrote here, but after a proper preparation and a case with at least a fraction of substance that has been put forward so far to make this happen
The way you are doing this is unnecessarily disruptive at the very least.
And from the position that I am in is simply disrespectful.
Finally, rpartial is a valid idea, but I have gone with this one.
Do you want me to discard this and you want to implement rpartial? What is your angle here? What are you trying to achieve?
In short, what do you want?
I'm not trying to disparage your work and would be happy if your PR gets into stdlib, but I also wanted to suggest another line of direction either as an alternative or for future work. Anyway, to reduce the noise on that PR, I'll mark my comments as off-topic.
The PR is looking pretty good at this point. Some of the variable names seem a bit gross, but that is minor. Also, I'm reconsidering whether having a "rough code equivalent" in the docs is still a good idea. For documentation purposes, a few informative example will likely communicate better than this snarl of code.
This PR needs a more detailed review than I can give it right now. @serhiy-storchaka Do you have time to comb through this one?
I did not follow the discussion on the Discourse, but is it settled? Are you fine with this, Raymond?
Before diving in the details:
-
partialmethod()should be updated as well. - The
inspectmodule should be updated to supportpartial()andpartialmethod()with a placeholder.
In the current version of the PR the pickle support is changed in incompatible way. If there are no placeholders, pickles should be compatible across versions.
Standardised variable naming a bit. Now it is as follows:
- without prefix or with
new_prefix means it is from callee arguments.new_is needed in C initializer asargswithout prefix includesfunc -
pto_prefix means it is ofpartial object. Which isfuncin initializer (if it ispartial) andselfin__call__. -
tot_prefix means it is total. I.e.pto_* + new_* -
nargskwmeans total number of arguments and keyword arguments. E.g.tot_nargskwin C__call__means final total number of arguments and keywords that will be sourced tofunc - Exposed variable for placeholder count is
placeholder_count. Non-exposed variable is namedphcount.
I think this improved general consistency and solved few issues, such as:
-
nargsvariable meant "number of arguments" in one place and "new arguments" in another. -
total_nargsmeantnargs + len(kw), while it somewhat suggested to benargs + pto_nargs.
It is not perfect, but I think it is a bit better now.
Factored out merging logic so it can be used for both partial and partialmethod.
Also, made them use the same __repr__. However, it turned out that they are not equivalent.
repr(partialmethod(func)) -> partialmethod(func, , )
while
repr(partial(func)) -> partial(func)
the one of partialmethod isn't very "nice".
There was only one test that needed fixing. However, if backwards compatibility is an issue here, let me know I will roll-back. I am not sure how much repr should be depended on.
I have a working and simplified implementation of https://github.com/python/cpython/pull/120783 adapted to this Placeholder version.
I can either update this PR with it or issue a separate one after. Whichever is more convenient.
Are you fine with this, Raymond?
Yes, I've exercised the API quite a bit and am fine with it. The core concept is sound, square = partial(pow, Placeholder, 2). This will address an irritating limitation without adding much cognitive or computing overhead.
All that is left is tweaking the PR to make it clean as possible before landing.
If you want to leave out partialmethod, I don't think that has to be done at the same time or at all. AFAICT it is very rarely used and doesn't warrant bogging down this one PR.
If you want to leave out
partialmethod, I don't think that has to be done at the same time or at all. AFAICT it is very rarely used and doesn't warrant bogging down this one PR.
It is done. Unless you don't like the way I did it?
ISTM that if a __get__ method were added to partial(), the new placeholder API would eliminate the need for partialmethod() or at least allow for a simpler implementation of it:
class Cell:
def __init__(self):
self._alive = False
@property
def alive(self):
return self._alive
def set_state(self, state):
self._alive = bool(state)
# Three ways of partialling methods
set_alive1 = partialmethod(set_state, True) # Works
set_alive2 = lambda self: self.set_state(True) # Works
set_alive3 = partial(set_state, Placeholder, True) # Needs Partial.__get__
The partial() class would need this new method:
def __get__(self, obj, objtype=None):
if obj is None:
return self
return MethodType(self, obj)
Potentially, the partialmethod() implementation could be simplified to something like this:
partialmethod = lambda func, *args, **kwargs: partial(func, Placeholder, *args, **kwargs)
At any rate, since set_alive2 works using lambda, people will expect that set_alive3 using partial would work as well.
I think it's important to maintain a mental model of partial(func, Placeholder, arg) as being equivalent to lambda x: func(x, arg). That would include the latter's descriptor behavior.
Now that the PR is maturing, performance should be checked for both CPython and PyPy. My expectation is that partial() with placeholders on CPython will be faster than an equivalent lambda. But on PyPy, the partial() with placeholders may be much slower than lambda and potentially should be avoided.
ISTM that if a
__get__method were added topartial()
This would simplify inspect.
Repr is not working well for partial (default "?" gets used):
MethodType(partial(f, 1), 1)
<bound method ? of 1>
https://github.com/python/cpython/blob/a905721b9c5c15279e67c2f7785034b7356b2d46/Objects/classobject.c#L282
Maybe partial can have __name__ and __set_name__?
Or leaving it to be "?" is ok for now?
ISTM that if a
__get__method were added topartial()
Also, maybe this can be left for separate PR?
As per suggestion of @serhiy-storchaka __repr__ of partialmethod will be fixed in a separate PR. Implementing your suggestion would mean it gets fixed in this PR. I am not sure how to go forward from here.
Would be good to get this merged and then apply the list of changes separately:
1.1. partialmethod.repr fix
1.2. partial.get and simplification of partialmethod
2.. kwds support in vectorcall and fallback removal
Also, maybe this can be left for separate PR?
Partialmethod doesn't need to be reimplemented in this PR but the __get__ method should be added to partial() as part of this PR. It is important to maintain the relationship with lambda.
This is very easy. Add the above 4 lines to the pure Python version. For the C version, just copy func_descr_get() from Objects/funcobject. I really don't want the behavior of partial and lambda to diverge beyond just pickling. The former needs to be explainable in terms of the latter. And whenever partial gets in the way of adding logging, type casts, validation, etc, then we want a plain def or lambda to be easily substitutable.
This is very easy.
It breaks several tests such as this: https://github.com/python/cpython/blob/a905721b9c5c15279e67c2f7785034b7356b2d46/Lib/test/test_inspect/test_inspect.py#L4283
They are easy to fix. Either:
- Add Placeholder
- wrap it in staticmethod
Nothing else in full test suite breaks, but it could break some external code.
Now that the PR is maturing, performance should be checked for both CPython and PyPy
Not sure how to benchmark PyPy. Never had anything to do with it. Would appreciate a bit of direction.
They are easy to fix. Either:
Add Placeholder wrap it in staticmethod
Either of those are fine. The test as written is incorrect. It is an example of Hyrum's law and tested what partial did do instead of what it should have done. We definitely want partial to behave like an equivalent def or lambda. The way people understand partial is by mentally unwinding it. As soon as Placeholder is added, it is certain that people will start using it to skip over self.
Adding __get__ to partial is a breaking change. It should be a separate issue and we should follow the common protocol for such changes: emit FutureWarning for few releases with suggestion to wrap partial into staticmethod, then change the behavior.
As for __repr__, I mean that it is better to fix it before merging this PR. This change could be backported.
We should not change pickle for Placeholder. It is not special enough, no other singleton except few builtins which are very special deserve this (and even for the latters there is now more general solution).