Weakly Referenceable Object

[nybblista]

Is there a public API that allows me to check whether an object is weakly referenceable? I know that you can check the value of the tp_weaklistoffset slot or check for the Py_TPFLAGS_MANAGED_WEAKREF flag to get the answer, but what about having something like:

• PyObject_WeaklyReferenceable or
• PyWeakref_Referenceable

or whatever name fits the situation? This would encapsulate things and keep the extension module developer away from the internal stuff.

https://discuss.python.org/t/weakly-referencable-object/80248

[colesbury]

Can you explain the use case a bit more? Why would you want to check if an object can have weak references?

[ZeroIntensity]

You could just check Py_TYPE(op)->tp_weaklistoffset != 0.

[nybblista]

My point is not about How? to check if an object is weakly referencable or not. My point is the method we use to check for a weakly referenceable object. To make my point clear, let's start with the PyWeakref_xxx APIs:

PyWeakref_Check
PyWeakref_CheckRef
PyWeakref_CheckProxy
PyWeakref_NewRef
PyWeakref_NewProxy
PyWeakref_GetRef
PyWeakref_GetObject
PyWeakref_GET_OBJECT
PyWeakref_IsDead
PyObject_ClearWeakRefs
PyUnstable_Object_ClearWeakRefsNoCallbacks

As we can see, we have functions that deal with the weak reference itself and functions that relate to the referent. However, when I start using the PyWeakref_xxx APIs for whatever case:

• Do I need to catch an exception from a call to PyWeakref_NewRef or PyWeakref_NewProxy to check that the referent is or is not a weakly referenceable object?

PyObject *number = PyLong_FromLong(10);
 
if(number != NULL){
  PyObject *weakrf = PyWeakref_NewRef(number, NULL);
  int result = 0;
  
  if(weakrf == NULL){
      if(PyErr_ExceptionMatches(PyExc_TypeError)){
          PyErr_Clear();
      }
  }else{
      result = 1;
  }
  
  printf("weakly referenceable?: %c\n", result? 'Y': 'N');
  Py_DECREF(number);
}

• Why do I need to change the pattern of my implementation to use tp_weaklistoffset or Py_TPFLAGES_MANAGED_WEAKREF (direct or indirect) to avoid the exception to check rather than keep everything consistent with the PyWeakref_xxx family?.

PyObject *number = PyLong_FromLong(10);
int result = 0;

if(number != NULL){
  if(PyType_HasFeature(Py_TYPE(number), Py_TPFLAGS_MANAGED_WEAKREF)
      || Py_TYPE(number)->tp_weaklistoffset != 0){

      PyObject *weakrf = PyWeakref_NewRef(number, NULL);
      result = 1;
      /* ... */
  }
  
  printf("weakly referenceable?: %c\n", result? 'Y': 'N');
  Py_DECREF(number);
}

• There are functions that serve the referent in the previous PyWeakref_xxx list that I have mentioned before, but why is there no API that tells me if an object is weakly referenceable?

if(PyWeakref_Referenceable(target)){
      PyObject *wref = PyWeakref_NewRef(target, NULL);
      /*...*/
}

• However, at the Python level, I know that there is more than one way to check for a weakly referenceable object, but why do I need to think this way?

import weakref
# ...
def is_weakly_referenceable(o):
    try:
        weakref.ref(o)
        return True
    except TypeError:
        return False

values = ["CPython", 3, {14}, UserObject()]

# How many exceptions should be thrown here?
wrvals = [ v for v in values if is_weakly_referenceable(v)]

• Can we make things more Pythonic by mapping PyWeakref_Referenceable to the weakref module as weakref.referenceable, for example?

import weakref
# ...
values = ["CPython", 3, {14}, UserObject()]
wrvals = [ v for v in values if weakref.referenceable(v)]

#...
if weakref.referenceable(item):
    pass

• At the CPython level, when we have a case that we want to check if the object is weakly referenceable without creating any weak reference, do we need to use tp_weaklistoffset and Py_TPFLAGS_MANAGED_WEAKREF again to check?

...
for(Py_ssize_t index = 0; index < items_count; index++){
    PyObject *item = PyList_GetItem(list, index);
    ...
    /* Is weakly referenceable? */
    result = (PyType_HasFeature(Py_TYPE(item), Py_TPFLAGS_MANAGED_WEAKREF)
    || Py_TYPE(item)->tp_weaklistoffset != 0);
            
    if(result){
        if(PyList_Append(wr_list, item) != 0){
            ...
        }
    }
    ...
}
...   

• What if the PyWeakref_Referenceable API is just an alias for the PyObject_WeaklyReferenceable API that encapsulates and hides the PyTypeObject internal stuff and use it for general situations like that?

#define PyWeakref_Referenceable    PyObject_WeaklyReferenceable

• Does this make our previous example hide the details of the PyTypeObject? Is it readable? Do we remove some overhead?

...
for(Py_ssize_t index = 0; index < items_count; index++){
    PyObject *item = PyList_GetItem(list, index);
    ...
    if(PyObject_WeaklyReferenceable(item)){
        if(PyList_Append(wr_list, item) != 0){
            ...
        }
    }
    ...
}
...   

Why do we need to use all of these details to check if an object is weakly referenceable if we can provide a clean, robust, and efficient interface to check for that?

• Encapsulate the check process and make it available from one place for all extensions developers, as PyObject_WeaklyReferenceable is better than leaving it for them to design theirs and to avoid using a lot of #ifdef in the future to make their extensions compatible with whatever update in PyTypeObject struct!
• I think adding PyWeakref_Referenceable as an alias for PyObject_WeaklyReferenceable will fill the gap for the referent objects in PyWeakref_xxx APIs.
• This will give the core developers more control over the implementation rather than let the extension developers use #ifdef here and there inside their modules in the future.
• One stable interface used by all extensions rather than multiple inconsistent interfaces.
• Keep the extension developers away from any internal changes to the PyTypeObject struct and its fields now and in the future.

I think if I have a reason for using PyWeakref_xxx APIs, for whatever situation, I have a reason to check for the referent if it is weakly referenceable or not, here or there.

I appreciate your time, and if Python supports something like this elsewhere, I am eager to learn/know.

[ZeroIntensity]

This seems like a documentation problem, if anything. It might be worth clarifying these things there, but I don't see the need for a new API. You should create an ideas thread for things like weakref.referenceable.

[nybblista]

@ZeroIntensity
I did not understand your point about "a documentation problem." Can you explain more about how this is a documentation problem? 👍🏻

[nybblista]

@colesbury
Does this help you to get the big picture of my point?

[ZeroIntensity]

I mean that this is a documentation problem in the sense that we don't have any note about detecting whether an object is weakly referenceable or not, I think adding that could be a good idea. But there's no reason we need a new API, because everything to detect that is already public.

If you're worried about "type object internals" (tp_weakreflistoffset is again, public, but let's go into a hypothetical situation with the limited API), then you could just check it the same way you would in Python: hasattr(obj, "__weakref__") or PyObject_HasAttrString(obj, "__weakref__"). If you really want, you could macro it and get your API:

#define PyObject_WeaklyReferenceable(op) PyObject_HasAttrString(op, "__weakref__")

[nybblista]

@ZeroIntensity I got it. Thank you for your explanation.

However, just because one API is public does not mean wrapping it with another public API is unnecessary. This can save us a lot of time and effort when the wrapped one is deprecated.

Regardless of the technical details, we have a good example here: the PyWeakref_IsDead API, which is added to 3.14. We can implement it as a macro and use the PyWeakref_GetObject, which is a deprecated public API, to check for a dead ref like:

#define PyWeakref_IsDead(op)  PyWeakref_GetObject(op) == Py_None

If we previously had this definition before the deprecation of PyWeakref_GetObject, for example, there is no need to fix whatever code that used the PyWeakref_GetObject to check for the dead refs in whatever code, and all we have to do is just redefine the PyWeakref_IsDead to be a function like what we have now(rather than asking the developers to update their implementations):

int
PyWeakref_IsDead(PyObject *ref)
{
    if (ref == NULL) {
        PyErr_BadInternalCall();
        return -1;
    }
    if (!PyWeakref_Check(ref)) {
        PyErr_Format(PyExc_TypeError, "expected a weakref, got %T", ref);
        return -1;
    }
    return _PyWeakref_IS_DEAD(ref);
}

However, we are unsure how other developers will "implement" their PyObject_WeaklyReferenceble. It could be with PyObject_HasAttrString, tp_weaklistoffset field, Py_TPFLAGS_MANGED_WEAKREF flag, or with catching an exception for PyWeakref_NewRef. And we are not sure which implementation of these will be deprecated.

So if we add a one-line-implementation like:

#define PyObject_WeaklyReferenceable   /* <any-implementation-here> */

This could help in the future if we meet a situation like PyWeakref_IsDead one day.

Again, regardless of the technical details, I think the motivation for PyWeakref_Referenceable and PyObject_WeaklyReferenceable is, somehow and to some extent, similar to the motivation of PyWeakref_IsDead except that the situation here is before, and there is after, a public API deprecation.

I hope that opens a window for any enhancement. I would be very happy if the result of the discussion of this issue were a new API, a weakref module function, a note in documentation, or whatever enhancement helps make Python better for us.

[ZeroIntensity]

PyWeakref_IsDead was a different case, I don't think it should be used as precedent here. Take a look at the issue: #128008

PyWeakref_IsDead was only added because the new function, PyWeakref_GetRef, could cause concurrent-deletion issues, and because it could call destructors--the internal _PyWeakref_IS_DEAD did not have that problem, so adding a new API actually provided a benefit. There's no such benefit here :(

However, we are unsure how other developers will "implement" their PyObject_WeaklyReferenceble. It could be with PyObject_HasAttrString, tp_weaklistoffset field, Py_TPFLAGS_MANGED_WEAKREF flag, or with catching an exception for PyWeakref_NewRef. And we are not sure which implementation of these will be deprecated.

FWIW, if we were to deprecate those, it would most likely be all at once. tp_weakreflistoffset is the offset of a __weakref__ attribute relative to the object's address, so if one goes, the other probably will. But we'll cross this bridge if we get there--I don't think a new API instead for some theoretical future deprecation will be a good pattern for us to follow.

But it's also not fair for me to be the only voice here. I'll tag @encukou who's quite more opinionated on the C API, because this kind of thing would need to go through the C API WG anyway.