Totals calculation

[ArcticCascade]

3.0.4 has a great update to calculating the totals.
I'm just curious as to why I would be seeing such a large difference between what cahcer and my server logs say.

2017-03-06 12:53:20.673 Since server start: 7.09 GB returned to clients, 1.46 GB stored from Internet, 0 bytes from peers; 0 bytes imported.

Cacher has retrieved the following stats for 2017-03-06:
4.8 GB of bandwith served to client devices.
1.4 GB of bandwith requested from Apple
0.0 bytes of bandwith requested from other Caching Servers

My server only runs between 6am and 11pm each day, so there's no crossover between dates.

[erikng]

What does "only runs between" mean? Are you limiting the cacher server via the network or some other means? The reason I ask is Cacher only calculates bandwidth served for the day and no longer calculates any data from the service itself. The log entry you posted is from the service, which could extend for days/weeks/months, up to the last time your server was rebooted. Thanks, Erik Gomez

…

________________________________ From: Joel Barker <notifications@github.com> Sent: Sunday, March 5, 2017 6:08:11 PM To: erikng/Cacher Cc: Subscribed Subject: [erikng/Cacher] Totals calculation (#35) 3.0.4 has a great update to calculating the totals. I'm just curious as to why I would be seeing such a large difference between what cahcer and my server logs say. 2017-03-06 12:53:20.673 Since server start: 7.09 GB returned to clients, 1.46 GB stored from Internet, 0 bytes from peers; 0 bytes imported. Cacher has retrieved the following stats for 2017-03-06: 4.8 GB of bandwith served to client devices. 1.4 GB of bandwith requested from Apple 0.0 bytes of bandwith requested from other Caching Servers My server only runs between 6am and 11pm each day, so there's no crossover between dates. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#35>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFa_GJGHbBs7aZjsfmbKBpQv7HIjp5pAks5ri2qLgaJpZM4MTpqo>.

[erikng]

Also, please try Cacher 3.0.3 and post what bandwidth it returns. It may be that the new function isn't calculating properly.

…

________________________________ From: Erik Gomez <e@eriknicolasgomez.com> Sent: Sunday, March 5, 2017 7:31:03 PM To: erikng/Cacher Subject: Re: [erikng/Cacher] Totals calculation (#35) What does "only runs between" mean? Are you limiting the cacher server via the network or some other means? The reason I ask is Cacher only calculates bandwidth served for the day and no longer calculates any data from the service itself. The log entry you posted is from the service, which could extend for days/weeks/months, up to the last time your server was rebooted. Thanks, Erik Gomez

________________________________ From: Joel Barker <notifications@github.com> Sent: Sunday, March 5, 2017 6:08:11 PM To: erikng/Cacher Cc: Subscribed Subject: [erikng/Cacher] Totals calculation (#35) 3.0.4 has a great update to calculating the totals. I'm just curious as to why I would be seeing such a large difference between what cahcer and my server logs say. 2017-03-06 12:53:20.673 Since server start: 7.09 GB returned to clients, 1.46 GB stored from Internet, 0 bytes from peers; 0 bytes imported. Cacher has retrieved the following stats for 2017-03-06: 4.8 GB of bandwith served to client devices. 1.4 GB of bandwith requested from Apple 0.0 bytes of bandwith requested from other Caching Servers My server only runs between 6am and 11pm each day, so there's no crossover between dates. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#35>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFa_GJGHbBs7aZjsfmbKBpQv7HIjp5pAks5ri2qLgaJpZM4MTpqo>.

[ArcticCascade]

From the server log:

2017-03-06 15:31:20.732 Since server start: 8.79 GB returned to clients, 1.95 GB stored from Internet, 0 bytes from peers; 0 bytes imported.

From cacher 3.0.3:

8.79 Gigabytes of bandwith served to client devices.

From cacher 3.0.4:

5.8 GB of bandwith served to client devices.

"Only runs between" means the server (the mac-mini) shuts down at 11pm every night and boots up again at 6am - so "Since server start" will always (excepting any reboots during the day) total from 6am that day.

[erikng]

Ping @Hunty1

[calum-github]

the new calculation method in my opinion is far more accurate.
It looks at every GET request from client devices and adds how much data was served to the client device.

I have no idea how apple was calculating the information from the "Since server start: "

Perhaps that includes dropped data, imported data, put data etc etc

[calum-github]

You would have to look at the logs for that period of time and see how apple comes up with that figure.

[erikng]

Hate to say it but I trust Apple more than us. There's something missing in the new commit. Thanks, Erik Gomez

…

________________________________ From: Calum Hunter <notifications@github.com> Sent: Sunday, March 5, 2017 9:29:37 PM To: erikng/Cacher Cc: Erik Gomez; Comment Subject: Re: [erikng/Cacher] Totals calculation (#35) the new calculation method in my opinion is far more accurate. It looks at every GET request from client devices and adds how much data was served to the client device. I have no idea how apple was calculating the information from the "Since server start: " Perhaps that includes dropped data, imported data, put data etc etc — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#35 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFa_GKL1ATrAC-N2wtB-2GINNlWJ1hszks5ri5nBgaJpZM4MTpqo>.

[calum-github]

Well it depends what figure you want. If you want the figures from the actual GET requests - the new commit is accurate.

@ArcticCascade you will need to attach the logs for the period you claim the difference in data.

Also doesn't 3.0.3 still do duplication on numbers because it looks at service logs as well as debug logs?

Cacher/cacher.py

Lines 848 to 853 in c7c4edd

	# Now combine all .log files in the destination into a temp file that's
	# removed when python exits
	rawLog = tempfile.TemporaryFile()
	for anyLog in glob.glob(os.path.join(tmpLogs, '*.log')):
	with open(anyLog, 'rb') as f:
	shutil.copyfileobj(f, rawLog)

My runs are showing the same data in metrics db 'bytes.fromorigin.toclients' as per cacher 3.0.4
So i'd probably trust that more than Since server start up

I've almost finished the work on adding an option to use either logs or the metrics db for measurement of bandwidth.

[ArcticCascade]

Thanks guys. I'll upload the logs when I get back to work tomorrow.

[erikng]

@Hunty1 while it is indeed duplicated, the first and last event will still be the correct data (according to since Server start). Regardless, let's wait.

@ArcticCascade please also post the db so we can compare.

[ArcticCascade]

Caching Logs.zip
Sorry for the delay. These logs should cover everything from March 6th.
I've included metrics.sqlite - I assume thats the db you mean.

[ArcticCascade]

I don't think there's anything wrong with your algorithm. I've taken a small sample from this morning's logs (Since server start: 369.4 MB) and added them up in Excel and come out with a total of 337.9 MB (matching cacher's results). Apple appears to be counting something else that's not logged in the usual way.

[calum-github]

Apple might be counting the HEAD requests