eZ Community » Forums » eZ Publish 5 Platform » Optimizing the in-memory Storage Backend
expandshrink

Optimizing the in-memory Storage Backend

Optimizing the in-memory Storage Backend

Saturday 15 October 2011 3:21:35 am - 4 replies

It seems that this backend stores the equivalent of every db table in an array, with elements being hashes.

All these arrays are indexed by a plain integer, so that every fetch function turns into a loop over the array checking until the item is found matching the desired id.

It would be probably much more efficient to make usage of the knowledge of the primary keys for every existing table, and use those as index into those arrays. Eg for objects we could index the "content" array by id, and for versions by content id + version nr, using a token as separator such as 2\\4. Then instead of loops, we would have direct access by array key to find version 4 of object 2, object 2 and node 2 (most frequent kind of accesses I think)

The management of the 'id' needs imho to be augmented: while it is good to have an autoincrement key, the possibility to have a primary key built out of many fields is more important.

Once we get this sorted out, adding a memcache layer as backing to the memory storage becomes suddenly more interesting...

We can make use of func_get_args() and late static binding if needed to create the equivalent of the old "fetch" fucntion that accepts as many parameters as there are inĀ  the definition of the primary key without having to be redefined in every php subclass.

ps: first post?

Saturday 15 October 2011 10:02:26 am

Hi Gaetano

Well, in-memory backend is here mainly for unit tests... It can probably be optimized, but the priority is much more given to the Legacy storage blunk.gif Emoticon

Saturday 15 October 2011 11:37:41 am

@GG: It was like this in the beginning, but I simplified the code and removed it as some data are looked up by several keys. The impact was 0% on the time unit tests use to run..
So if you want to speed something up, lookinto Storage/Legacy/Tests/Content/SearchHandler/_fixture/full_dump.php and the use of that..

Sunday 16 October 2011 4:53:31 pm

I fully agree that legacy storage engine is the first priority.

But a storage where data is indexed by the most-used key can imho be used as proof-of-concept / example for all key-value store implementations. As long as we are not going into such complexity that it could be called an in-memory-database...

I am not sure unit test suite is the best way to measure performances of the storage layer btw, unless you have a unit test where you load a couple million objects into the store: doing 100000 loops on a 100 elements array is not the same as doing 100 loops on a 100000 elements array...

Monday 17 October 2011 10:41:10 am

My point is: We don't intend to make that piece of code more complex as long as it does not give us (API) any benefits, after all we would then need to maintain it. It's not intended to be a generic engine, and we have bigger fish to fry.

expandshrink

You must be logged in to post messages in this topic!

36 542 Users on board!

Forums menu

Proudly Developed with from