Out of memory with high ReadBuffer value?

[mint]

We are running a Delphi webservice which provides customer report data. The data is refreshed every 30 minutes. At some point an EOutOfMemory exception occurs at the customer.

We could reproduce this on a NT4 machine when the webserver ran for about 15 hours (=30 refreshs). The strange thing is that there was plenty of free RAM left at that time (one customer reported it happened to him when there were 1.8 GB of physical RAM available).

I have a stack trace from the point when the problem occured on our NT4 test machine. After opening the TOracleDataset it stopped in GetMem call with EOutOfMemory. The code executed before looks like the following. The redundancies and unnecessary checks are due to the fact that I put the code of four methods together for this posting.

Qry := TOracleDataSet.create(nil);
Qry.Session := FDOASession;
Qry.SQL.Text := ASQL;
if Qry.Active then Qry.Close();
Qry.SQL.Text := ASQL;
Qry.ReadOnly := True;
Qry.ReadBuffer := 10000;

 

[Marco Kalter]

The ReadBuffer is a tuning property. It controls how many records are transferred to the application in one client/server roundtrip. Setting it too high (and 10,000 is really too high) will decrease performance because of memory overhead. 25 us a good default, and for performance critical queries you can set it to 100 or 200, depending on the record size.

Note that the ReadBuffer does not affect the maximum size of the actual result set.

------------------
Marco Kalter
Allround Automations

 

[TheTimo]

Hello,

we have the same problem and we need a large buffer size, because we transfer a lot of data (for one query about 10000 to 60000 records) a lot of time in a hour (about 1000 to 3000). When we set the buffer too low, the transfer is much too slow, because we let run the application on 5 computer at the same time. So what can we do ?

Oliver Schubert

 

[Marco Kalter]

Setting the buffer size so high that it causes memory problems will not improve performance. It needs to be much, much lower.

Perhaps you can let me know the performance difference between a ReadBuffer of 25, 100 and 1000? Can you also let me know if you are using a TOracleQuery to fetch these 60000 records?

------------------
Marco Kalter
Allround Automations

 

[mint]

Unfortunately this problem is still not resolved. I have set the ReadBuffer to 150. Still I see an OutOfMemory exception after about one day.

The situation has improved though because the refresh cycle for testing has been set to 10 times/hour, not 2 times/hour as before and the application still holds up for about a day. However this indicates that the OutOfMemory problem is related directly to DOA. This theory is corroborated further by the following callstack:

user name : administrator
operating system : Windows 2000 Service Pack 2 build 2195
system language : German
physical memory : 454

 

[Marco Kalter]

This would suggest a memory leak, which may of course be caused by several things. In no particular order:

- Direct Oracle Access
- Oracle Net
- Delphi/C++Builder's VCL
- Other 3rd party components
- Your application

The fact that the system runs out of memory when the TOracleDataSet attempts to allocate a record block is in no way an indication that this is also the cause of the problem. It's merely a symptom.

Perhaps you can use Memory Sleuth or some other memory leak detection utility to find out where the memory leaks are?

 

[mint]

I agree that this smells like a memory leak. However as there is plenty of RAM still available (and the machine is still usable normally otherwise) this leak must hit a certain (buffer?) limit. If the application would have a mem leak I'd expect that all of the RAM (incl. virtual memory) would have been used up... I try and see what MemProof has to say.