ATLAS-5055: Incremental Export : When entity exported has a tag propa… - #377
Conversation
…gated from entity which is deleted , tag is not propagated to it at target
pareshddevalia
left a comment
pareshddevalia
left a comment
There was a problem hiding this comment.
LGTM code perspective
|
|
||
| for (AtlasEdge propagationEdge : propagationEdges) { | ||
| if (getEdgeStatus(propagationEdge) != ACTIVE) { | ||
| if (getEdgeStatus(propagationEdge) != ACTIVE && !deferredActionEnabled && !RequestContext.get().isImportInProgress()) { |
There was a problem hiding this comment.
@chaitalicod - with addition of && !deferredActionEnabled here, tags from deleted entities will be propagated even when import is not in progress. Note that the default value of AtlasConfiguration.TASKS_USE_ENABLED is true, which will result in the if at line 704 evaluate to false, hence will continue with propagation. This does not look correct. Can you please validate?
There was a problem hiding this comment.
@chaitalicod - when deferredActionEnabled is true, this if condition will evaluate to false, hence will continue to propagate classification - even when getEdgeStatus(propagationEdge) is not ACTIVE i.e. deleted edge. This doesn't seem correct.
There was a problem hiding this comment.
Hello @mneethiraj ,
Sorry for late response I was working on this comment I have changed my approach and created a PR
#434
Kindly review
3 participants
…gated from entity which is deleted , tag is not propagated to it at target
What changes were proposed in this pull request?
Create lineage table1 --> table2 ---> table3 ---> table4
Add tag tag1 to table1
tag1 is propogated till table4
Drop table1
Export and Import entire lineage along with parent using incremental fetch type
CURRENT BEHAVIOUR:
table1 , table2 , table3 are exported
But since table1 is deleted , on the destination after import tag appears to be applied on table1 but tag is not propagated to any of the child entities.
Root cause:
when deferred action is enabled and such a case if this is imported from source to target on target side the propagation doesn't happen
Flow with deferred actions enabled:
classification is created
entities are created
Add propagation task is created
classification is mapped to parent
Because parent entity is expected to be in deleted state it goes and deletes entities
After this the Add propagation task attempts to propogate the classification but fails as it finds the edge deleted between parent and classification
Patch fix : Add a check for ignore if import in progress and deferred actions is enabled
for a code block in getimpactedvertices where we check for status of edge if not active it doesn’t propagate the tag
How was this patch tested?
Sanity testing and by importing multiple entities with ctas and parent is deleted it works successfully