How to access reverse direct relations through sdk

Hi @Gargi Bhosale ,

You cannot get the reverse direct relations using the .retrieve or .list methods, you have to use the .query method. Typically, you do have to use the `NodeResultExpression` with `node.through` set to the direct relation property of the view on the other side of the reverse direct relations and the `nodes.direction=’inwards’`. 

An alternative is to use `pygen`. `pygen` generates a tailor made SDK for your data model. In the generated, `.list` method there is a parameter `retrieve_connections` that you can use to retrieve connections. This will do the `.list` or `.query` call for you. See https://cognite-pygen.readthedocs-hosted.com/en/latest/usage/listing_filtering_retrieving.html#reading-instances-list-and-retrieve

I see that pygen is highly specific on a particular model. In my case ,I intend to make the code generic to any model the user would select. In that case, would you recommend using the .query method? Are there any performance differences between the both?

There are two cases.

1. If the direct relation of the other side is a single direct relation, use the query method. client.data_modeling.instances.query(...) You can see an example in the pygen example here.
2. If the direct relation of the other side is a list of direct relations, use the search method.
   client.data_modeling.instances.search(...). You will do a lookup on the nodes with the direct relation filtering for matching the nodes with the reverse direct relation.
    

The pygen example for client.data_modeling.instances.query(...) is not very clear and what I am trying is not working.

this is what I am trying can you please correct?

from cognite.client.data_classes.data_modeling.query import Query, Select, NodeResultSetExpression, EdgeResultSetExpression, SourceSelector
view=ViewId(space='slb-pdm-dm-governed', external_id='Property', version='1_7')
nested_prop_view_id = ViewId(space='slb-pdm-dm-governed', external_id='PropertyType', version='1_6')
properties = ["externalId", "propertyType"]
nested_prop_view_properties=["externalId"]

query= Query(
    with_={"PropertyType":NodeResultSetExpression(
                        from_="Property",
                        direction="inwards",
                        filter= HasData(views=[nested_prop_view_id]),
                        through=nested_prop_view_id.as_property_ref("directSingle"),
                    )},
    select={"PropertyType": Select([SourceSelector(nested_prop_view_id, nested_prop_view_properties)])}
)
res1 = client.data_modeling.instances.query(query)

Another important observation: 

when I use client.data_modeling.instances.search(...) without the query because I want everything and with limit= None, I get only 100 records, whereas I have much more than them. When I increase the limit to 1000 it gives 1000 records, I have more than 1000 as well. Ideally, I expect all the data to come when I pass limit=None.

Give me a few days, and I will create a full example in the SDK docs on how to do this.

Sure Anders, let me know when it is available

I have published it temporarily here: https://cognite-pygen.readthedocs-hosted.com/en/latest/docs-dev/docs_query.html

Thank you, so when I use the search() for reverse direct relation of listable relation, I see that we get 1000 instances by using client.data_modeling.instances.search(activity_view, filter=is_equipment, limit=1_000)

I am not getting any cursor or any other way to retrieve the full data, all the instances satisfied with the query, can you please help me with this?

​@Anders Albert  - We still are not able to get the complete data with the temporarily approach or query suggested ? Do you see any code change is required in sdk or some other way of querying is required.

We are blocked as of now due to this.

Appreciate if you can expedite your response here.

Regards,
Neeraj B

​@Gargi Bhosale Search do not support pagination. I have extended the example to show how you can do a lookup on each individual equipment instead to ensure you get all activities: https://cognite-pygen.readthedocs-hosted.com/en/latest/docs-dev/docs_query.html#limitation-with-pagination

Hello ​@Anders Albert , 

I cannot make sense what is written in the link. I assume there is a problem in formatting, could you please give the example in a better format so that I can understand? :)

Seems the documentation was broken. Should be fixed now. Can you try again?

https://cognite-pygen.readthedocs-hosted.com/en/latest/docs-dev/docs_query.html#limitation-with-pagination

Hello Anders,

I read the documentation. It seems that we are assuming in the example that one equipment can have maximum 1000 activities only. Two observations on this from our side:

1. In our use case we can have more than 1000 instances for one equipment.
2. Filtering approach will not be performant as we process a lot of data in our application.

We want to understand:

1. Any strong reason for not supporting this in query SDK.
2. If the above feature is not possible for some reason, for a performant solution we would need support for pagination in search.

This is a limitation on the server side with the /search endpoint. If you want to do an exhaustive, you have the use the /list endpoint, which is implemented in the `.list(...)` method in the SDK.

This is a trade-off. Search is high-performant good for fetching specific items fast. The `.list()` is slower but supports pagination (baked in to the SDK) so you can do an exhaustive listing.  

Hi ​@Anders Albert ,

Say you have a reverse direct relation: 

Equipment <---- Activities

That is,

Activities has a relation to Equipment  and other properties defined as

equipment:[Equipment]
property1: String

and

Equipment as a reverse direct relation to Activities defined as

activity: [Activities]
@reverseDirectRelation(throughProperty: "equipment")

How would you list all instances in Equipment filtered on properties in Activities?

I tried something like (i.e. tried to traverse the relation) but this times out :

client.data_modeling.instances.list(
        instance_type="node",
        sources=EQUIPMENT_VIEW_ID,
        filter=Nested(
                 scope=EQUIPMENT_VIEW_ID.as_property_ref("activity"), 
                 filter=In(ACTIVITY_VIEW_ID.as_property_ref('property1'), ["x", "y"])
               ),
        limit=None
        )

This works if the relation is a normal relation (i.e. going the other way from Activites to Equipment), but not from Equipment to Activities.

Secondly, because of this seeming limitation, are there benefits for having reverse direct relations?

You do the lookup on the activity and not the equipment. 

Something like this

client.data_modeling.instances.list( instance_type="node", sources=ACTIVITY_VIEW_ID, filter=In(ACTIVITY_VIEW_ID.as_property_ref('property1'), ["x", "y"]), limit=None )

Hi ​@Anders Albert ,

Apologies, I see now my question set up was incomplete.

Say, Equipment and Activities are defined as below: 

type Equipment
  @view(version: "1")
{
  property1: String
  activity: [Activities]
    @reverseDirectRelation(throughProperty: "equipment")
}


type Activities
  @view(version: "1")
{
  property2: String
  equipment: Equipment
}

What’s the best way to retrieve Equipment instances filtered on property1 (Equipment property) and property2 (Activities property)?

Something like this times out:

client.data_modeling.instances.list(
        instance_type="node",
        sources=EQUIPMENT_VIEW_ID,
        filter=And(
                 In(EQUIPMENT_VIEW_ID.as_property_ref("property1"), ["a", "b"],
                 Nested(
                 scope=EQUIPMENT_VIEW_ID.as_property_ref("activity"), 
                 filter=In(ACTIVITY_VIEW_ID.as_property_ref('property2'), ["x", "y"])
                 )
               ),
        limit=None
        )

I wrote up an article to try to address your issue. I think the issue is likely lack of indices on the properties you are filtering on, causing the timeout. Workaround is to try to split the query into two

I published it temporarily here: https://cognite-pygen.readthedocs-hosted.com/en/latest/docs-dev/query_filter.html