Data Preprocessing Inquiry for NODLINK Algorithm

[cycloneha]

Hello! I found your work very interesting.
I tried to generate some test data myself using Ubuntu OS and sysdig. The command I used was sudo sysdig -j > sysdig_output.json. However, I found that the data I generated cannot be correctly read by the NODLINK algorithm. Specifically, NODLINK prompts me that the input data is missing some key columns. I would like to know if you have preprocessed the data, and if it would be possible for you to share your preprocessing method or explain your data processing approach.
Thank you!

[0xllssFF]

Hello! I found your work very interesting. I tried to generate some test data myself using Ubuntu OS and sysdig. The command I used was sudo sysdig -j > sysdig_output.json. However, I found that the data I generated cannot be correctly read by the NODLINK algorithm. Specifically, NODLINK prompts me that the input data is missing some key columns. I would like to know if you have preprocessed the data, and if it would be possible for you to share your preprocessing method or explain your data processing approach. Thank you!

The command line I used to collected the data is as follows:

sudo sysdig -p "*%evt.num %evt.time %evt.type %evt.args %fd.name %proc.pid %proc.name %proc.cmdline %proc.ppid %proc.pname %proc.pcmdline" "proc.name!=sysdig and ((evt.dir= > and (evt.type=read or evt.type=readv or evt.type=write or evt.type=writev or evt.type=fcntl or evt.type=recvfrom or evt.type=sendto or evt.type=sendmsg or evt.type=recvmsg)) or (evt.dir = < and (evt.type=rename or evt.type=accept or evt.type=clone or evt.type=rmdir or evt.type=chmod or evt.type=execve or evt.type=pipe or evt.type=fork)) ) and (not fd.name exists or fd.name != '')" -j > data.json

[hancoding1024]

Hello!
It seems that I have encountered a similar problem. The difference is that I use Windows system and ETW, and I use four providers: Microsoft-Windows-Kernel-File, Microsoft-Windows-Kernel-Network, Microsoft-Windows-Kernel-Process, and Microsoft-Windows-Security-Auditing. The log data I collected also cannot be correctly processed by the algorithm because of the lack of some key attributes, especially the 'CommandLine' attribute. I would like to know if you can share the log collection method or tool on the Windows system?
Thank you very much!

[0xllssFF]

Hello! It seems that I have encountered a similar problem. The difference is that I use Windows system and ETW, and I use four providers: Microsoft-Windows-Kernel-File, Microsoft-Windows-Kernel-Network, Microsoft-Windows-Kernel-Process, and Microsoft-Windows-Security-Auditing. The log data I collected also cannot be correctly processed by the algorithm because of the lack of some key attributes, especially the 'CommandLine' attribute. I would like to know if you can share the log collection method or tool on the Windows system? Thank you very much!

You can refer to this: https://github.com/microsoft/perfview/blob/main/documentation/TraceEvent/TraceEventLibrary.md. Hope it can help you write the collection scripts.