String fileds vs Memory usage

[Daniel Stoch]

Hello,

Why DOA in TOracleDataSet allocates full memory for string fields?

For example I declare two TStringFields: NAME with size 100 and DESCRIPTION with size 200; all records in DataSet uses all memory for these fields (it doesn't matter how many characters these fields contain for each record).
So when I declare these fields 100/200 long, DataSet uses (it's example) 3068KB, but with sizes 50/100 it uses "only" 1644KB. Even if all records have empty strings in these fields.

Is there any reason that DOA does not support dynamic memory allocation for strings? When value in record for string field with Size=100 has 15 characters, then it should using memory for 15 characters, not for all 100 chars.

Best regards,
Daniel Stoch

 

[Marco Kalter]

I believe this is the nature of the TDataSet and TField classes. It is all based on fixed size buffers.

 

[Daniel Stoch]

Sorry but I don't agree with you. It is possible to use dynamic size buffers with TDataSet and TField classes. What does it mean "I believe"? Probably you are not sure. It is very strange for me: you write such components as DOA and you are not sure how does VCL classes are implemented and how to implement dynamic size buffers?!!!

Sorry for this example, but ODAC components from CoreLab uses dynamic size buffers, so as you can see it is possible. Your DOA components are very good, but dynamic size buffer can make them much better .

 

[Daniel Stoch]

Hello,

Could someone reply to my previous post? Or maybe you are implementing "dynamic" string buffers now?

Best regards
Daniel Stoch

 

[Marco Kalter]

I looked into this, and it is indeed possible to implement a feature to compress the dataset result set storage. The record buffer needs to comply to TDataSet/TField rules, but one can store the entire result set as one sees fit. Maybe we can add this in a future release.

 

[Jens Fudickar]

Hi Macro,

is it possible to say more then maybe?

I think this is a very very helpfull feature for people working with large datasets and large varchar fields, so i would realy appreciate if we can see a 4.1 in January or February 2004 with this feature

Greetings
Jens

 

[Marco Kalter]

For read-only datasets this is relatively simple to implement. For updateable datasets we have to rewrite a large part of the memory management. We have to look into this before I can say anything more specific than "maybe".

 

[Daniel Stoch]

Hi,

Thanks for your answer Marco and thank you Jens for joining this topic. I think it would be great if you will be able to implement this feature. In our application we process huge amount of data, so effective memory management is a very important for us.

Good luck,
Daniel

 

[al0]

Originally posted by Daniel Stoch:
Hi,

In our application we process huge amount of data, so effective memory management is a very important for us.

I dont't want to meddle with your discussion, but cost of intensive memory allocation/deallocation for dynamic buffer managment may (or may not) heavily overweight benwefits from memory saving.

Yours,

Oleksandr

 

[Jens Fudickar]

You may be right.

Because of this reason it should be an optional parameter.

Greetings
Jens Fudickar

 

[Daniel Stoch]

Originally posted by al0:
I dont't want to meddle with your discussion, but cost of intensive memory allocation/deallocation for dynamic buffer managment may (or may not) heavily overweight benwefits from memory saving.

Yours,

Oleksandr

Hi,

It depends on situation. For example when you only browsing data (and cache data on client application) from many SQL queries, I think the cost should not be such big. Another thing is that now computers are quite fast (CPU and memory), so cost of dynamic alloc/dealloc should not overweight benwefits from memory saving (IMHO ).

And as Jens said it should be an optional parameter (both for read-only and modifiable TOracleDataSet).

Best regards,
Daniel

 

[al0]

Originally posted by Daniel Stoch:

It depends on situation. Another thing is that now computers are quite fast (CPU and memory), so cost of dynamic alloc/dealloc should not overweight benwefits from memory saving (IMHO ).

And as Jens said it should be an optional parameter (both for read-only and modifiable TOracleDataSet).

Definitely, it depends, so I have written "it
may (or may not) overweight ...".

Really, memory allocation maintaince is quite time-consumming process (especially if you have a lot of memory chunks) and borland memory manager is not the best (while it is not hte worst as well). And all this stuff are intended to really large sets we (I guess) million of records. So performance degradation may be noticable. BTW, if datasets are really big it is not very wise to fetch the whole dataset to the client. And I can't even imagine browsing of such amount of data. Reporting? May be, but who need a report with handred thousand rows? And filtering (normally) has to be performed on server. So I'm not sure that any programm really need this dynamic memory management for dataset (IMHO much more likely that program that claim the need in this feateru has serious design flaw).

Yours,

Oleksandr

 

[Jens Fudickar]

I'm not sure, that you are completly right.

For example. A table has 20 Varchar2-Fields with Length 4000. This means 80000 Bytes for one record.
1000 Records means 80.000.000 Bytes.
And i think 1000 Records is not so much.
We have sometimes tables with some very large fields, which are mostly completly empty.
In such cases there is a lot memory to save which is not used.

Greetings
Jens

 

[al0]

Originally posted by Jens Fudickar:
I'm not sure, that you are completly right.

For example. A table has 20 Varchar2-Fields with Length 4000. This means 80000 Bytes for one record.
1000 Records means 80.000.000 Bytes.
And i think 1000 Records is not so much.
We have sometimes tables with some very large fields, which are mostly completly empty.
In such cases there is a lot memory to save which is not used.

Greetings
Jens

Possibly, but for such structure I would consider CLOB instead of varchar2. And fetch them on client only on request.

Yours,

Oleksandr

 

[Jens Fudickar]

You are principle right with using clobs. But there are some disadvantages of clobs.

And there are old database structures using varchar2. So it's not allways possible.

Greetings
Jens

 

[Daniel Stoch]

Originally posted by Jens Fudickar:
I'm not sure, that you are completly right.

For example. A table has 20 Varchar2-Fields with Length 4000. This means 80000 Bytes for one record.
1000 Records means 80.000.000 Bytes.
And i think 1000 Records is not so much.
We have sometimes tables with some very large fields, which are mostly completly empty.
In such cases there is a lot memory to save which is not used.

Greetings
Jens

Exactly, this a good example.

Another one: we browse in our application data from many diffrent selects. And all results are cached on client (for reducing unneeded quering of DB). Then if these selects contain data from Varchar2 fields, application can very fast eat 100MB of system memory. And it is not necessary to fetch 1000000 records, 100-1000 per one select is enough to allocate all of system memory.

And I don't agree with you Oleksandr that "IMHO much more likely that program that claim the need in this feateru has serious design flaw).". There are some specific situations when it is necessary.

 

[al0]

Originally posted by Daniel Stoch:
And I don't agree with you Oleksandr that "IMHO much more likely that program that claim the need in this feateru has serious design flaw).". There are some specific situations when it is necessary. [/QB]

You are right that specific situation may exist. But it still more likely that program has design flaw then that situtation is really so specific.
In opposite case situation will be not specific but quite common

 

[Jens Fudickar]

Marco,

did you have any new informations on this issue?

Is it possible to realize this only for the readonly datasets? This can be a lot of help for some parts!!

Greetings
Jens

 

[Marco Kalter]

It's on the list of enhancement requests, but I don't have a concrete date.

 

[Johan]

Maybe you can publish a list with requested enhancements on this site.

This list would contain the requested enhancements and an indication when/if it will be implemented. (or something like that)

Greetings Johan Visser

 

[Jens Fudickar]

Hi Marco,

i think this is a realy good idea.

Greetings
Jens

 

[Jens Fudickar]

Hi Marco,

Maybe you can ask your customers which of the features they want to see next .

Greetings
Jens

P.S. For me the memory problem is realy important

 

[Holger Dors]

Jens, couldn't you use TOracleQuery (which, I believe, needs less memory) and copy the results into a memory table which supports compression, like TkbmMemTable?

As you're talking about read-only datasets, I thought that this could be an option for you...

 

[Jens Fudickar]

Holger,

that's not possible. The overhead is too big.

For example. With a dataset there is no effort to load only 1000 records and on demand the next 1000. Doing this tOracleQuery and MemDatasets you got a lot of work.

Greetings
Jens

 

[Daniel Stoch]

Hi,

Copying the results into a memory table takes some time. Holger, your solution for me sound like: "why don't you create a new TDataSet descendant class, inherited form TkbmMemTable, which will be using TOracleQuery for fetching records" . It looks like a new implementation of TOracleDataSet! I know it is possible, we have in our application cached DataSets based on TkbmMemTable (which use TOracleQuery) for querying lookups' values, but they don't give us all functionality TOracleDataSets do (eg. QBE).

But to modify records, we can first copy them into memory table (eg. TkbmMemTable). But for browsing only, it takes to much time (for implemementing this too). So dynamic size buffers for read-only TOracleDataSet could be a great feature .

Best regards,
Daniel

 

[Jens Fudickar]

Yep!!!

Greetings
Jens

 

[Daniel Stoch]

Hi,

Is there any chance for implementing this functionality, described in this thread over one year ago?
Or is it, like people says in my country, "the dream of cut head"?

Best regards,
Daniel Stoch

 

[Marco Kalter]

This turned out to be a little bit more work than expected, but it is still on our to-do list. Maybe we can add this in 4.1.

 

[Jens Fudickar]

4.1 ??

Any timeframe, list of planed changes or so ?

Greetings
Jens

 

[Marco Kalter]

Not yet.

 

[Andrey Maximenko. Ukraine]

Marco,
did you have any new informations on this issue?
-----------------
Desc for any field, based on User string Function, return size 4000:
select myStringFunction() as test from dual:
desc: test VARCHAR (4000)
-----------------------
Greetings

 

[Marco Kalter]

This did not make it into 4.1, which focuses on Unicode support. We'll look into this for a 4.1 patch release.

 

[Andrey Maximenko. Ukraine]

up

 

[Nashev]

Is it still in TODO?
It's really memory-eating behavior, and because it in our selects all string functions surrounded with substr(function(), 1, 255). But it's not good solution, more good solution is to make strings in TOracleDataset dinamically-allocated. We waiting it!