Transaction Logging Query

[robm82]

Problem Description

We are designing a point-in-time recovery solution using archive-style transaction logging and the native Domino Backup task.

Completed transaction-log extents are written to a dedicated local T: drive. We intend to run load backup -t hourly and use the Domino Backup DB config to store the completed extents to an Azure File Share. Rubrik will separately protect the Azure File Share.

HCL Support has confirmed that, when the configured Backup Translog Command succeeds, the Domino Backup task marks the extent as successfully archived. If the command fails, the extent remains pending and is presented again during the next load backup -t run.

HCL Support also advised that, after acknowledgement, Domino can recycle or delete archived transaction logs automatically and that there is no separate manual Domino command for recycling them.

We would like clarification from the Domino Backup developers about the exact lifecycle of these extents and the resulting disk-space behaviour on the live transaction-log drive. HCL Support have asked us to post the information here to get direct assistance from the backup developer.

Specifically:

1. When the native File Backend successfully copies a completed transaction-log extent, does the Domino Backup task call NSFDoneArchivingLog for that extent?
2. Once the extent is archived and no longer required for restart recovery, does the Domino logging subsystem:
   • delete the physical .TXN file and return space to the operating system;
   • retain and reuse/rename the existing physical file; or
   • choose between these behaviours depending on internal conditions?
3. If extents are retained for reuse, should administrators expect used space on the transaction-log drive to plateau rather than decrease?
4. Is the recycling/deletion timing configurable, or is it managed entirely by the Domino logger?
5. Is there a supported command, statistic, log message, or API for monitoring:
   • extents waiting to be archived;
   • extents marked as archived;
   • archived extents still active for restart recovery;
   • extents eligible for recycling; and
   • the current archive backlog?
6. Is manual deletion of an archived .TXN file ever supported, or should all lifecycle management be left exclusively to the Domino logger?
7. Does the native Domino Backup File Backend provide all the required Domino Backup API interaction when the destination is an ordinary network file share, without requiring the destination storage or Rubrik to be Domino-aware?

This clarification is required so that we can size and monitor the dedicated transaction-log drive correctly and determine whether successful hourly archive runs will prevent indefinite disk growth.

HCL Support Ticket

HCL Support Ticket Number and status: CS1448802

Status: HCL Support provided initial guidance and directed us to the Domino Backup GitHub repository for clarification from the backup developers.

HCL Support advised that:

Once an archived transaction-log extent has been successfully protected and acknowledged, Domino can recycle or delete it automatically. There is generally no separate manual Domino Backup command for recycling archived logs.

Environment

Domino Version

Exact Domino Server version, including Fix Pack and Hotfix:

14.5.1 HF43

Operating System

Platform and exact version:

• Windows Server 2022
• Running as an Azure IaaS VM

[Daniel-Nashed]

First of all I would expect that support would send this type of request to development instead asking you to open an issue on GitHub.
This project is mainly intended to help with Domino Backup Integrations.

Domino Backup related functionality itself, clarification about the functionality and details you are asking for should come thru HCL support/DEV.

By coincident I used to be the lead developer of Domino Backup.
And I am still involved in integrations with other products and write up use cases etc for the community side in this GitHub project.

First of all let me explain that Domino Backup leverages the Domino C-API as documented in the C-API User Guide.
This is the official interface Domino provides for any third party backup vendor and uses it exactly like this for Domino Backup.
This also includes the handling of transaction logs in archive translog style.

The API is really the interface to mark transaction logs archived. And Domino itself re-uses translog extends (renames them instead recreates them).
The reason behind renaming the files is to avoid creating a new file (more I/O).

There is some logic to keep a certain amount of translog extends for re-use and there are some notes.ini parameter to control the behavior.
But the general functionality makes sure the translog extends can be re-used.

This makes disk space monitoring a bit more challenging.
Therefore Domino Backup added statistics and notes.ini variables which are updated at the end of each archive translog backup.

I perfectly understand where you are coming from and what your concerns are.
By coincident I am looking into archive translog in a customer situation for a >8000 user environment with 4 larger mail servers right now.
I added statistics to the Domino Prometheus compatible end point DomProm to cover translog in more detail.
And we are leaving those additional stats in combination with the Domino Backup stats.

The most important sentence for me is your last statement. It helps me to understand the intend of your technical detailed questions.

** This clarification is required so that we can size and monitor the dedicated transaction-log drive correctly and determine whether successful hourly archive runs will prevent indefinite disk growth. **

Let me try to answer your questions. An hourly backup is a good schedule. I would not schedule anything lower.
The Domino backup log should provide you with details about how much translog was generated in this time.
But it is difficult to find out how much translog is really needed. This depends on your data and it might be also a matter of configuration.

I am currently comparing different servers and fine tuning translog in this 8k users customer environment.
For example disabling translog in mail.boxes
You would probably love my Garafana dashboards I just wrote last week after adding the stats to DomProm.

• Domino Backup leverages the NSFDoneArchivingLog () call exactly how it is documented in the C-API -- See initial link
• Domino itself is responsible for deleting the translog extend. If you run into any issues this should be covered by the ticket
• Not all translogs are deleted. They are recycled and Domino takes track of it
• I will outline all statistics in Domino and the Domino Health Monitor Stats I added to the DomProm application.
• There are no separate stats for wait, marked archived, etc. But see my info further down what can be done.
• You should not manually delete translog .txn files as part of your operations
• If there is something wrong and you are sure they have been backed-uped, you can delete them if really needed. This does not harm the translog functionality
• But they have to have been marked as archived!
• Domino Backup provides an interface to backup NSF and TXN files. There is no specific API involved. In the File Integration scenario this is a standard file copy operation.

Domino Backup Stats

The stats available give you an idea about the last translog extend which is in backup.

• There can be new translog extends that are not yet complete
• And there can be pre-allocated translog extends not even used
• There should not be more than 64 extra TXN files out of the box

You should plan reasonable amount of translog to at least survive a day or two that your backup is not working.

I would monitor the number of TXN created. We are using Prometheus data in Grafana to track translog.
Let me write another reply with details.

This isn't part of the product but a free offering which can help you when you use Grafana.
I am mainly wrote the DomProm integration for Linux. But it would be also available on Windows.

DominoBackup.LastBackup.TL.LastLogExtend = S0006169.TXN
DominoBackup.LastBackup.TL.Status = Successful
DominoBackup.LastBackup.TL.Time = 07/15/2026 17:24:31 CEDT

Domino Health Stats (domprom)

This isn't included in Domino but is an open source offering.
Check details about the project Domino Grafana and DomProm specifically.

# HELP DominoHealth_translog_file_count Number of current Transaction Log Extends in Translog directory 
# TYPE DominoHealth_translog_file_count gauge 
DominoHealth_translog_file_count 65 
# HELP DominoHealth_translog_file_max Highest Transaction Log Extend Number in Translog directory 
# TYPE DominoHealth_translog_file_max gauge 
DominoHealth_translog_file_max 259793 
# HELP DominoHealth_translog_file_min Lowest Transaction Log Extend Number in Translog directory 
# TYPE DominoHealth_translog_file_min gauge 
DominoHealth_translog_file_min 259729 
# HELP DominoHealth_translog_style Domino Statistic Database.RM.Sys.Log.Type 
# TYPE DominoHealth_translog_style gauge 
DominoHealth_translog_style 1 

I hope this is helps you? Is there anything you feel that should be turned into a GitHub pages document?
Your questions are very specific. But the intend behind is something customers using archive translog will often have to look into.
Specially when there is a backup share involved.

In our case we are planning to backup to a central ZFS share as the first tier. And this includes also translog.
Translog data would be a good compression candidate. I don't have numbers yet. But I will write up something as soon we have the backup in place.

[Daniel-Nashed]

Here is an example stat. I am calculating the average increase over a day. I have the same also hourly, for every 2 weeks and per month.
This helps me to see if my translog optimization helps.

For 2 of the 4 servers I disabled translog on mail.box and log.nsf. It look like we got a least 20% less translog.

green = mail01a
yellow = mail01b

blue = mail02a
orange = mail02b

The second servers in the cluster got mail.boxes and log.nsf translog disabled.
Next step will be to only disable mail.box translog on the two other servers to see the difference log.nsf makes.

Grafana helps to monitor and we also have alerting and check the current translog extends.
Those servers have 500 GB translog but hardly use it.
Translogs are cleaned up in time. But always around 64 files should remain for reuse after translog backup.

Note: divided by 16 because each TXN is 64 MB and we want to convert it to GB.

[Daniel-Nashed]

Here is an example how translogs look on 4 larger production servers.
You can see that we always have 64 translog extends after translog archive did ran. This is normal and not an issue. I added the stats to the DominoHealth stats in DomProm exactly for this purpose to monitor them
We have added alerting for the file-system free space (which is another part of DominoHealth stats).

There was never a problem to clean those files up. I could show you stats for a couple of month.
But I am showing you 7 days here to make it better readable.

[Daniel-Nashed]

I have some questions for you, about your integration.

• Where are you storing your translogs and backups exactly? On a file server?
• For how long are you storing your backups there and how do you backup to Rubrik?
• If this is a tiered backup, how long do you keep the data on the remote storage location?
• How do you make sure you are not overflowing that disk space? Is this deduplicating storage?

I wrote this concept document some days ago, because I am looking into implementing it for a customer --> https://opensource.hcltechsw.com/domino-backup/domino-backup-concept/

We are going to leverage ZFS because it has multiple advantages

• Deduplicating file-system --> incremental storage for backups
• Compression --> some of the backup data, specially translogs can be compressed quite well

ZFS is one good way which would also work with Windows using a Windows file-share.
The same principle would work with other dedup solutions. Just coping the whole data to a normal volume would generate quite some disk usage if you keep the backup for a while on the disk location.

If you move the backup to a Rubrik backup and not keep it longer than a day, how do you restore?
You would need restore integration.

In our concept we plan to keep like 30 days of backup on ZFS for NSF and probably 2 weeks of translogs for point in time recovery.
The full backup is planned to be scheduled once per night.

I am curious how you are running your backup.

[Daniel-Nashed]

I think Domino native statistic makes make sense. I have added an AHA idea you might want to vote on:

https://domino.ideas.hcl-software.com/ideas/DOMINO-I-3234

I could add it to DomProm but this would needs to use the backup API to iterate over the translog extends without marking them as done.
But my tests have shown that this API should not be used twice. That means it would be eventually in conflict with the backup solution.
Domino itself should write those stats.
If you have additional stats you would like to see, please respond in the AHA idea.

[robm82]

Hi Daniel,

To add some further context about what I am trying to design (it is by no means finalised yet, and still has to undergo some testing!). Also, I am not a Domino Admin expert - I am bit of a "Jack of all trades, master of none", so if some of my terminology is incorrect I do apologise :).

Where are you storing your translogs and backups exactly? On a file server?

We are storing the Translogs on a local disk with Azure Premium SSD Storage, and the idea is that we will be configuring the Domino and the Backup DB to store hourly backups/archives of the Transaction Logs to an Azure File Share.

For how long are you storing your backups there and how do you backup to Rubrik?

The plan is to keep 7 days worth of Transaction Logs on the Azure File Share. Unfortunately, Azure Files does not have a native mechanism for retention policies, so I am thinking we may need to script this. But I may see if we can use Azure Blob, as we can then get retention policies to auto-clear old trans log archives!

If this is a tiered backup, how long do you keep the data on the remote storage location?

The plan is to have it as hot storage

How do you make sure you are not overflowing that disk space? Is this deduplicating storage?

Because this is an Azure Storage Account, not too worried about the disk space overflowing

If you move the backup to a Rubrik backup and not keep it longer than a day, how do you restore?
You would need restore integration.

We haven't really thought too much (yet) about the restore procedure, but the initial thoughts we had were to restore the transaction log archives back to the same Azure Storage Account, but to a restored folder. But to be honest, our plan is to only go back to a point in time for the past 7 days anyway, so Rubrik backing up the Azure Storage Account is more of a "belt-and-braces" approach to make sure if the data is corrupt or unavailable in the Storage Account, we can still get it back with Rubrik.

Thanks for all your input so far Daniel - it's really valuable, and you seem to really know what you are talking about 😄

[Daniel-Nashed]

Thanks for the details and indicating my answers help you.
I am a consultant in the field working with customers directly on integrations.

Azure Storage isn't cheap. If it is not de-duplicating every backup is the full set of data.
That's why I am asking and that's why I am proposing a de-duplicating option.
ZFS is a very good option and it would also work for Windows servers. You would just need a Linux host with Azure standard storage.
But there might be also other options for deduplication.

For Domino backup and file-system target the Domino Backup takes care of all retention for NSF and Translog if configured.
The default config that ships with dominobackup.ntf has the right configuration for NSF and Translog for (backup, restore, retention).

The backup concept I linked works with primary backup, snapshot for protection and optional 2nd backup tier. this can be almost anything.
In our case this will likely be Proxmox backup. in your use case this is Rubrik.
We are going to have 30 days for NSF restore and probably 2 weeks of translog point in time. Older restores would allow to restore the daily backup.

The first tier restore does not need any additional integration. It just works out of the box.

In all type of file-backend integrations the databases are consistent. Delta files are automatically merged.
This means the 2nd tier backup to Rubrik can be copied back if needed.
There would be ways to automate the restore. But if this is more for DR, I would not automate that restore.
Your primary backup would just work without any additional scripting.

Please let me know what other concerns you have and continue to brainstorm more what makes sense in your environment.
There is no one size fits all. It really depends on what you have available in your environment.

-- Daniel