DT read on empty file

[RuchiDhanuka]

Hello Rob

When I do a dt read on empty preexisting file, dt never throws data miscompare. Is this by design and any reason to keep it so?
Is there a way to catch the miscompare or failure in this situation? Any help is appreciated!

[RobinTMiller]

Hi Ruchi,
I don't understand, did you say an "empty" file?
How can dt report a miscompare if there's nothing to read? A read of an empty file returns an end of file indication, which is handled as such by dt.

What am I missing?

What is the dt command line and reported statistics?
Please add enable=debug to acquire more information.

Thanks,
Robin

[RuchiDhanuka]

Hey Rob

Thanks for responding quickly.
Assuming a scenario as below:
dt writes data, then assume a SW bug truncates the file to 0, later dt read silently passes — Is there a way to handle this case where the dt read can reliably throw an error because we know that data was initially written by dt successfully

DT log with debug does indicate EOF. should we consider "End of file detected," as an indicator of failure?
Can we rely on debug mode with EOF all the time? Also, when we set a limit in the dt params during read, should dt ensure that the data read matches the limit stated. ?

20260420 092859 00m00.00s (dt j:0 t:0): Cloning device 0xb4e2a0
20260420 092859 00m00.05s (dt j:0 t:0): Directory: /t/ip/fg1_8645/, File: /t/ip/fg1_8645//p5nqilkp5nqilk0, Base Name: p5nqilkp5nqilk0
20260420 092859 00m00.05s (dt j:0 t:0): Mounted from device: [ip]:/fg1
20260420 092859 00m00.05s (dt j:0 t:0): Cloning device 0xb579c0 - /t/ip/fg1_8645//p5nqilkp5nqilk0...
20260420 092859 00m00.05s (dt j:0 t:0): The Monitor Thread ID is 0x7f5cc1bd3700
20260420 092859 00m00.01s (dt j:1 t:1): Starting I/O, Job 1, Thread 1, Thread ID 0x7f5cc1a90700
20260420 092859 00m00.01s (dt j:1 t:1): Opening input file /t/ip/fg1_8645//p5nqilkp5nqilk0, open flags = 040000 (0x4000)...
20260420 092859 00m00.01s (dt j:1 t:1):
20260420 092859 00m00.01s (dt j:1 t:1): File Name: /t/ip/fg1_8645//p5nqilkp5nqilk0
20260420 092859 00m00.01s (dt j:1 t:1): File Inode: 9272807230690820196 (0x80afa10c00000064)
20260420 092859 00m00.01s (dt j:1 t:1): Directory Inode: 9272807213510950976 (0x80afa10800000040)
20260420 092859 00m00.01s (dt j:1 t:1): File Size: 0 (0)
20260420 092859 00m00.01s (dt j:1 t:1): Operation: open
20260420 092859 00m00.01s (dt j:1 t:1): I/O Mode: read
20260420 092859 00m00.01s (dt j:1 t:1): I/O Type: sequential
20260420 092859 00m00.01s (dt j:1 t:1): File Type: input
20260420 092859 00m00.01s (dt j:1 t:1): Direct I/O: enabled (bypassing cache)
20260420 092859 00m00.01s (dt j:1 t:1): Desired Access: 0x4000 = O_DIRECT
20260420 092859 00m00.01s (dt j:1 t:1):
20260420 092859 00m00.01s (dt j:1 t:1): Input file /t/ip/fg1_8645//p5nqilkp5nqilk0 successfully opened, fd = 3
20260420 092859 00m00.00s (dt j:1 t:1): Record #1 - Reading 262144 bytes (512 blocks) into buffer 0x7f5cc1a92000, lba's 0 - 511 (offset 0)
20260420 092859 00m00.01s (dt j:1 t:1): File name: /t/ip/fg1_8645//p5nqilkp5nqilk0
20260420 092859 00m00.01s (dt j:1 t:1): End of file detected, count = 0, error = 0 [file #1, record #1, read 0 file bytes, 0 total bytes]
20260420 092859 00m00.01s (dt j:1 t:1): Closing file /t/ip/fg1_8645//p5nqilkp5nqilk0, fd = 3...
20260420 092859 00m00.01s (dt j:1 t:1): I/O has completed, thread exiting with status 0...
20260420 092859 00m00.01s (dt j:1 t:1):
20260420 092859 00m00.01s (dt j:1 t:1): Operating System Information:
20260420 092859 00m00.01s (dt j:1 t:1): Host name: host1 (ip)
20260420 092859 00m00.01s (dt j:1 t:1): Process ID: 9211
20260420 092859 00m00.01s (dt j:1 t:1): OS information: Linux 4.18.0-553.70.1.el8_10.x86_64 #1 SMP Tue Aug 12 13:09:16 EDT 2025 x86_64
20260420 092859 00m00.01s (dt j:1 t:1):
20260420 092859 00m00.01s (dt j:1 t:1): File System Information:
20260420 092859 00m00.01s (dt j:1 t:1): Mounted from device: [ip]:/fg1
20260420 092859 00m00.01s (dt j:1 t:1): Mounted on directory: /t/ip/fg1_8645
20260420 092859 00m00.01s (dt j:1 t:1): Filesystem type: nfs4
20260420 092859 00m00.01s (dt j:1 t:1): Filesystem options: rw,relatime,vers=4.1,rsize=1024,wsize=1024,namlen=255,hard,proto=tcp6,timeo=600,retrans=2,sec=sys,clientaddr=ip,local_lock=none,addr=ip
20260420 092859 00m00.01s (dt j:1 t:1): Filesystem block size: 1024
20260420 092859 00m00.01s (dt j:1 t:1):
20260420 092859 00m00.01s (dt j:1 t:1): Total Statistics:
20260420 092859 00m00.01s (dt j:1 t:1): Input device/file name: /t/ip/fg1_8645//p5nqilkp5nqilk0 (device type=regular)
20260420 092859 00m00.01s (dt j:1 t:1): Type of I/O's performed: sequential (forward, rseed=0x69e62a1b00070ec9)
20260420 092859 00m00.01s (dt j:1 t:1): Job Information Reported: Job 1, Thread 1
20260420 092859 00m00.01s (dt j:1 t:1): Data pattern string used: 'IOT Pattern' (blocking is 512 bytes)
20260420 092859 00m00.01s (dt j:1 t:1): Last IOT seed value used: 0x01010101
20260420 092859 00m00.01s (dt j:1 t:1): Total records processed: 0 with min=512, max=262144, incr=variable
20260420 092859 00m00.01s (dt j:1 t:1): Total bytes transferred: 0 (0.000 Kbytes, 0.000 Mbytes)
20260420 092859 00m00.01s (dt j:1 t:1): Average transfer rates: 0 bytes/sec, 0.000 Kbytes/sec
20260420 092859 00m00.01s (dt j:1 t:1): Number I/O's per second: 0.000
20260420 092859 00m00.01s (dt j:1 t:1): Number seconds per I/O: 0.0000 (0.00ms)
20260420 092859 00m00.01s (dt j:1 t:1): Total passes completed: 1/1
20260420 092859 00m00.01s (dt j:1 t:1): Total errors detected: 0/1
20260420 092859 00m00.01s (dt j:1 t:1): Total elapsed time: 00m00.01s
20260420 092859 00m00.01s (dt j:1 t:1): Total system time: 00m00.00s
20260420 092859 00m00.01s (dt j:1 t:1): Total user time: 00m00.00s
20260420 092859 00m00.01s (dt j:1 t:1): Starting time: Mon Apr 20 09:28:59 2026
20260420 092859 00m00.01s (dt j:1 t:1): Ending time: Mon Apr 20 09:28:59 2026
20260420 092859 00m00.01s (dt j:1 t:1):
20260420 092859 00m00.01s (dt j:1 t:1): Thread exiting with status 0...
20260420 092859 00m00.01s (dt j:1 t:1): END OF TEST
20260420 092859 00m00.01s (dt j:1 t:1): Test explicitly passed
20260420 092859 00m00.01s (dt j:1 t:1): Thread 0x7f5cc1a90700 is exiting...
20260420 092859 00m00.07s (dt j:0 t:0): Cleaning up device 0xb5d920, master 0...
20260420 092859 00m00.07s (dt j:0 t:0): Cleaning up device 0xb579c0, master 0...
20260420 092900 00m00.67s (dt j:0 t:0): Thread 0x7f5cc2d93700 is exiting...
20260420 092900 00m01.07s (dt j:0 t:0): Terminating with exit code 0...
20260420 092900 00m01.07s (dt j:0 t:0): Exiting with status 0...
20260420 092900 00m01.07s (dt j:0 t:0): Program is exiting with status 0...
20260420 092900 00m01.07s (dt j:0 t:0): END OF TEST
20260420 092900 00m01.07s (dt j:0 t:0): Test explicitly passed
20260420 092900 [.] END OF TEST / RESULTS SUMMARY
20260420 092900 [.] No results reported

[RuchiDhanuka]

Write was done before. Read performed after sometime.
Read command used
dt iobehavior=dt enable=debug bs=64k pattern=IOT max=256k incr=var noprogt=30 noprogtt=40 flags=direct iotype=sequential if=/t/ip/fg1_8645/p5nqilkp5nqilk0 limit=100M

[RobinTMiller]

Thanks for the dt command line!

Assuming dt wrote the full 100m earlier, if dt was able to read the full 100m, the exit status will be zero (0). But adding enable=eof will cause dt to exit with a status of 254 to indicate EOF/EOM was encountered.

Other than using the exit status, you would need to parse the read information from the statistics. The format of the statisics (roll your own) can be specified via the tkeepalive= option, if desired.

Hope this helps, Cheers Robin

[RuchiDhanuka]

When i tried using dt read enable=debug,eof options on a truncated file with limit=100m (previously written 100mb), but it did not exit with status 254. Am i missing anything?

my dt version
--> Date: September 21st, 2023, Version: 25.05, Author: Robin T. Miller <--

dt limit=100m if=./f2 enable=debug,eof iobehavior=dt pattern=incr dispose=keep
00m00.00s dt (j:0 t:0): Cloning device 0x9a12a0
00m00.02s dt (j:0 t:0): Device size: 512, Real Device Size: 512, User Device Size: 512
00m00.02s dt (j:0 t:0): Device size: 512, Real Device Size: 512, User Device Size: 512
00m00.02s dt (j:0 t:0): Directory: ., File: ./f2, Base Name: f2
00m00.02s dt (j:0 t:0): Mounted from device: ip:/vol1
00m00.02s dt (j:0 t:0): Cloning device 0x9ac130 - ./f2...
00m00.02s dt (j:0 t:0): The Monitor Thread ID is 0x7f6b477b4700
00m00.01s dt (j:1 t:1): Starting I/O, Job 1, Thread 1, Thread ID 0x7f6b46a12700
00m00.02s dt (j:1 t:1): Opening input file ./f2, open flags = 0 (0)...
00m00.03s dt (j:1 t:1):
00m00.03s dt (j:1 t:1): File Name: ./f2
00m00.03s dt (j:1 t:1): File Inode: 9248704677384608182 (0x8059ffe70000d9b6)
00m00.03s dt (j:1 t:1): Directory Inode: 9248704518470762560 (0x8059ffc200000040)
00m00.03s dt (j:1 t:1): File Size: 0 (0)
00m00.03s dt (j:1 t:1): Operation: open
00m00.03s dt (j:1 t:1): I/O Mode: read
00m00.03s dt (j:1 t:1): I/O Type: sequential
00m00.03s dt (j:1 t:1): File Type: input
00m00.03s dt (j:1 t:1): Direct I/O: disabled (caching data)
00m00.03s dt (j:1 t:1): Desired Access: 0 = O_RDONLY
00m00.03s dt (j:1 t:1):
00m00.03s dt (j:1 t:1): Input file ./f2 successfully opened, fd = 3
00m00.00s dt (j:1 t:1): File name: ./f2
00m00.00s dt (j:1 t:1): End of file detected, count = 0, error = 0 [file #1, record #1, read 0 file bytes, 0 total bytes]
00m00.00s dt (j:1 t:1): Closing file ./f2, fd = 3...
00m00.00s dt (j:1 t:1): I/O has completed, thread exiting with status 0...
00m00.01s dt (j:1 t:1):
00m00.01s dt (j:1 t:1): Operating System Information:
00m00.01s dt (j:1 t:1): Host name: host1
00m00.01s dt (j:1 t:1): User name: root
00m00.01s dt (j:1 t:1): Process ID: 3624
00m00.01s dt (j:1 t:1): OS information: Linux 4.18.0-553.117.1.el8_10.x86_64 #1 SMP Fri Mar 27 18:49:45 EDT 2026 x86_64
00m00.01s dt (j:1 t:1):
00m00.01s dt (j:1 t:1): File System Information:
00m00.01s dt (j:1 t:1): Mounted from device: ip:/vol1
00m00.01s dt (j:1 t:1): Mounted on directory: /t/mnt
00m00.01s dt (j:1 t:1): Filesystem type: nfs4
00m00.01s dt (j:1 t:1): Filesystem options: rw,relatime,vers=4.2,rsize=65536,wsize=65536,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=ip,local_lock=none,addr=ip
00m00.01s dt (j:1 t:1): Filesystem block size: 65536
00m00.01s dt (j:1 t:1): Filesystem free space: 976609869824 (931367.750 Mbytes, 909.539 Gbytes, 0.888 Tbytes)
00m00.01s dt (j:1 t:1): Filesystem total space: 1044506345472 (996118.875 Mbytes, 972.772 Gbytes, 0.950 Tbytes)
00m00.01s dt (j:1 t:1):
00m00.01s dt (j:1 t:1): Total Statistics:
00m00.01s dt (j:1 t:1): Input device/file name: ./f2 (device type=regular)
00m00.01s dt (j:1 t:1): Type of I/O's performed: sequential (forward)
00m00.01s dt (j:1 t:1): Job Information Reported: Job 1, Thread 1
00m00.01s dt (j:1 t:1): Data pattern read: 0x03020100 (incrementing 0-255)
00m00.01s dt (j:1 t:1): Total records read: 0
00m00.01s dt (j:1 t:1): Total bytes read: 0 (0.000 Kbytes, 0.000 Mbytes, 0.000 Gbytes)
00m00.01s dt (j:1 t:1): Total records written: 0
00m00.01s dt (j:1 t:1): Total bytes written: 0 (0.000 Kbytes, 0.000 Mbytes, 0.000 Gbytes)
00m00.01s dt (j:1 t:1): Total records processed: 0 @ 512 bytes/record (0.500 Kbytes)
00m00.01s dt (j:1 t:1): Total bytes transferred: 0 (0.000 Kbytes, 0.000 Mbytes)
00m00.01s dt (j:1 t:1): Average transfer rates: 0 bytes/sec, 0.000 Kbytes/sec, 0.000 Mbytes/sec
00m00.01s dt (j:1 t:1): Number I/O's per second: 0.000
00m00.01s dt (j:1 t:1): Number seconds per I/O: 0.0000 (0.00ms)
00m00.01s dt (j:1 t:1): Total passes completed: 1/1
00m00.01s dt (j:1 t:1): Total errors detected: 0/1
00m00.01s dt (j:1 t:1): Total elapsed time: 00m00.00s
00m00.01s dt (j:1 t:1): Total system time: 00m00.00s
00m00.01s dt (j:1 t:1): Total user time: 00m00.00s
00m00.01s dt (j:1 t:1): Starting time: Tue Apr 28 13:43:33 2026
00m00.01s dt (j:1 t:1): Ending time: Tue Apr 28 13:43:33 2026
00m00.01s dt (j:1 t:1):
00m00.01s dt (j:1 t:1): Thread exiting with status 0...
00m00.01s dt (j:1 t:1): Thread 0x7f6b46a12700 is exiting...
00m00.06s dt (j:0 t:0): Cleaning up device 0x9b50c0, master 0...
00m00.06s dt (j:0 t:0): Cleaning up device 0x9ac130, master 0...
00m00.06s dt (j:0 t:0): Terminating with exit code 0...
00m00.06s dt (j:0 t:0): Exiting with status 0...
00m00.06s dt (j:0 t:0): Program is exiting with status 0...

[RobinTMiller]

Sorry about that, the eof flag is no longer being honored. Historic flag, I have not used for many years.

Nonetheless, you can check for an empty file outside of dt, or use a total keepalive to reports stats interested in, then parse and check accordingly. When I find time, I'll see why the eof flag is no longer honored. I originally added a flag since most automation did not handle the EOF exit status of 254, rather expecting 0 for success and non-zero for failure.

\robin@LAPTOP-BJH5MV95 ~/GitHub/dt/windows/x64/Release
$ ./dt if=emptyfile bs=64k limit=1m pattern=iot tkeepalive="Records read: %R" stats=brief
dt.exe (j:1 t:1): Records read: 0

robin@LAPTOP-BJH5MV95 ~/GitHub/dt/windows/x64/Release
$

Note: The keepalive strings are user defined, please see "dt help" for valid format control strings.

[RobinTMiller]

Please update and rebuild dt to pickup the fix for exiting with an EOF (254) exit status, thanks!

robin@DESKTOP-SBC6MG3:/mnt/c/cygwin64/home/robin/GitHub/dt/linux-ubuntu22$ ./dt version
--> Date: April 29th, 2026, Version: 26.2, Author: Robin T. Miller <--
robin@DESKTOP-SBC6MG3:/mnt/c/cygwin64/home/robin/GitHub/dt/linux-ubuntu22$ ./dt if=emptyfile bs=64k records=1 enable=edebug,eof
dt (j:1 t:1): File name: emptyfile
dt (j:1 t:1): End of file detected, count = 0, error = 0 [file #1, record #1, read 0 file bytes, 0 total bytes]
dt (j:1 t:1):
...
dt (j:0 t:0): Program is exiting with status 254...
robin@DESKTOP-SBC6MG3:/mnt/c/cygwin64/home/robin/GitHub/dt/linux-ubuntu22$ echo $?
254
robin@DESKTOP-SBC6MG3:/mnt/c/cygwin64/home/robin/GitHub/dt/linux-ubuntu22$
~

Note: Only Ubuntu22 and Windows executables are prebuilt, due to limited build machines available nowadays.

[RuchiDhanuka]

Thank you very much Rob for fixing it very quick. I shall try this out.

[RobinTMiller]

Usually when reading an existing file, dt will use the size of the file or the user data limit if smaller. This logic kept dt from encountering an end of file (EOF) condition. Thus this change:
While reading an existing file, allow enable=eof to honor the user data limit for reading past EOF thus forcing an EOF exit status.

robin@DESKTOP-SBC6MG3:/mnt/c/cygwin64/home/robin/GitHub/dt/linux-ubuntu22$ ./dt version
--> Date: April 30th, 2026, Version: 26.3, Author: Robin T. Miller <--
robin@DESKTOP-SBC6MG3:/mnt/c/cygwin64/home/robin/GitHub/dt/linux-ubuntu22$ ./dt of=dt.data bs=64k limit=1m pattern=iot dispose=keep disable=stats
robin@DESKTOP-SBC6MG3:/mnt/c/cygwin64/home/robin/GitHub/dt/linux-ubuntu22$ ./dt if=dt.data bs=64k limit=1m pattern=iot disable=stats enable=edebug,eof
robin@DESKTOP-SBC6MG3:/mnt/c/cygwin64/home/robin/GitHub/dt/linux-ubuntu22$ ./dt if=dt.data bs=64k limit=1m+b pattern=iot disable=stats enable=edebug,eof
dt (j:1 t:1): File name: dt.data
dt (j:1 t:1): End of file detected, count = 0, error = 0 [file #1, record #17, read 1048576 file bytes, 1048576 total bytes]
dt (j:0 t:0): Program is exiting with status 254...
robin@DESKTOP-SBC6MG3:/mnt/c/cygwin64/home/robin/GitHub/dt/linux-ubuntu22$

The above logic will allow detection of files truncated with less data than originally written via the EOF (254) exit status.