Attention to decrease performance

[MasiumDev]

I enable this plugin and it decreases the performance of other queues.
Do you test it on high publish about 20,000/sec?

[hmaiga]

Hi @MasiumDev,
Do you have some feedbacks after few months of usage i believe.
I would like to use it but i'm a bit worried about the performance of how messages are kept in cache.

[VSilantyev]

I think performance is O(n) from number of unique items in the queue.

Seems like there is linked list for holding cache entities:

rabbitmq-message-deduplication/lib/rabbitmq_message_deduplication/cache.ex

Line 189 in b2227b7

case cache |> Mnesia.read(entry) |> List.keyfind(entry, 1) do

List.keyfind:
https://hexdocs.pm/elixir/1.13/List.html

[noxdafox]

Mnesia read always returns a list entries as a result. Therefore, we need to find the entry within the list. As the list is either containing no entries (cache miss) or one entry (cache hit), the big-O cost is O(1)(Mnesia table lookup) + O(1)(one-size list lookup).

The reason this issue is still open is to give an example to the community on how not to write a ticket on an open source project. Commenting someone's work as "your code is slow, do you even test it at scale?" is not contributing in any way to improving things.

The OP is not providing any meaningful information such as:

• The expected behaviour
• The observed behaviour
• How to reproduce the issue
• Platform configuration (OS, software versions, plugin version)
• Affected resources (queues, nodes, exchanges, ...)

A minimum knowledge in the domain would suggest that implementing a caching layer over a distributed service will indeed incur in computing costs. As multiple publishers might be pushing the same message at the same time onto different RabbitMQ nodes, we need to guarantee a replicated and transactional storage for the deduplication headers. This obviously does not come for free. Hence, a performance impact is to be expected.

The real question is whether the impact caused by this plugins is significantly higher than alternative solutions implemented in similar ways. If there would be a benchmark showing that, for example, a set of consumers relying on a Redis cluster for deduplication would show significantly higher consumption throughput then, and only then, there would be a valid concern to be raised.

[VSilantyev]

@noxdafox Thank you for such detailed reply. Indeed I was wrong about O(n), missed a part about Mnesia.read. My bad, I do not know erlang.
It would be great to add your explanation to readme. Should I create a PR?

[noxdafox]

@VSilantyev don't worry I was not addressing you in the answer but rather highlighting the issue with the overall ticket.

I am not sure these architectural details are much of help for the reader of a README. In there, usually a reader expects to find:

• Why is this thing existing ---> What problem it solves
• How to use this thing ---> Installation and configuration
• How to approach the community ---> Here is some improvement I'd like to do myself

The "how this thing works" and "when you should not use this thing" are usually more on the technical documentation/wiki side.
I have few ideas on providing usage examples and more in depth wikis but my time is currently a bit limited.

[y-nk]

coming here long time after, i guess we still need to do:

If there would be a benchmark showing that, for example, a set of consumers relying on a Redis cluster for deduplication would show significantly higher consumption throughput then, and only then, there would be a valid concern to be raised.

to know if that's a concern to be addressed?