File handle leaks

Oct 12, 2011 at 5:06 PM

I am using your assembly in a multi-threaded app and one issue I have found is that after a Pst object goes out of scope and is garbage collected the file handle is still held by the app. I was able to alleviate this condition somewhat by uncommenting the code in the Dispose methods for the classes Pst, Folder, Message, etc... But it seems the iterators for Recipients and Attachments do not clean themselves up when they have gone out of scope.

I am guessing you have already struggled with this as indicated by the commented code in the Dispose methods. If you have an idea about where the issue lies I would gladly research it and hopefully come up with a fix.

 

Developer
Oct 12, 2011 at 5:50 PM

I've figured out the issue about a month ago, but just haven't had time to write a proper fix for it.  The issue actually lies in the fact that all of the dispose methods being commented, obviously, as that wouldn't free any file handles.  The reason they are commented is that once the GC actually hits the disposer and finalizer, it will sometimes delete the shared pointer to the pst object, causing any classes you still have open to become invalid.  Now, this doesn't happen all of the time, but when it does, it's a pain, worse yet, it's seemingly random when it does happen, this largely due to the fact that the GC is non-deterministic.

However, the solution is fairly easy, what needs to be done in the C++/CLI code is that the SharedDBPointer needs to keep track of its own references.  While the situation isn't ideal, it's probably the most logical to get this problem solved.  Each time the SharedDBPointer is copied to another class, (ie a PropertyBag or TableContext), the reference should be increated, and each time a delete is called in SharedDBPointer, it should be decremented, and if 0, should be deleted.  Of course, templates like this actually exist in C++, so I've been wanting to look into how easily they could be used in a mixed-managed environment, but like I said, I haven't had the time to delve into that yet.  If this problem can be solved, however, the Dispose/Finalizers can be fixed and the software can work like it should.

 

Thanks,

Christopher

Oct 13, 2011 at 2:42 PM

Thanks for the quick reply.

An additional data point - if I uncomment the code in the Dispose methiods and do not use Message.Recipients or Message.Attachments I do not leak handles. With these changes I can at least dump the messages in a pst to HTML or txt.

Developer
Oct 13, 2011 at 5:17 PM

Ah, that should help me pinpoint the issues.  I spent some time on it yesterday and got quite far, more than I expected.  I'm working on fixing the main issue, which is early finalization of certain classes.  I've only been able to get an exception thrown in debug, not release.  I am not sure if that is because the issue is less prevalent in release vs debug, or worse yet, if release just isn't throwing an exception (this exception happens during the GC calling the Finalizer of the class).

I'm going to run a few more tests today, but if all goes well, I should be able to push updated code.

Developer
Oct 13, 2011 at 10:01 PM

Also, I found the issue with the disposing not working properly for the Recipients or Attachments.  They both use the underlying Table class, which was missing a pointer deletion.  If you add this code:

 

if(_table != nullptr)
            {
                delete _table;
                _table = nullptr;
            }

It will solve that problem.  That fix is currently in a separate branch, where I'm trying to re-work how the native memory is managed between all of the classes.  I got further in depth on the issues we're having with this at this SO question

Please note, however, uncommenting the deconstructors right now may have disastrous effects on your code, specifically that the GC is finalizing the objects too early.  Your code may fail in some cases, particularly when you are enumerating at the same time that a garbage collection occurs.

 

Oct 13, 2011 at 10:13 PM

I already see that slice of code in !TableEnumerator.

Developer
Oct 17, 2011 at 9:31 PM

If you pull the latest source, all of the memory leaks should be fixed.  This changeset is the most stable we've had yet!

Oct 17, 2011 at 10:23 PM
I am testing it right now will inform you of my results.

On Mon, Oct 17, 2011 at 4:31 PM, ccurrens <notifications@codeplex.com> wrote:

From: ccurrens

If you pull the latest source, all of the memory leaks should be fixed. This changeset is the most stable we've had yet!

Read the full discussion online.

To add a post to this discussion, reply to this email (pstsdknet@discussions.codeplex.com)

To start a new discussion for this project, email pstsdknet@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com


Oct 18, 2011 at 8:33 PM
Good job overall. I have found a few cases where the file handle remains open after disposing of all the objects and performing a GC. The code below is my test case:
IPst rdopststore = new Pst(some_pstfile);
IMessage msg = rdopststore.OpenMessage((pstsdk.definition.util.primitives.NodeID)some_nodeid);
if (msg != null)
{
foreach (pstsdk.definition.pst.message.IAttachment rdoattachment in msg.Attachments)
{
using (var bw = new System.IO.BinaryWriter(System.IO.File.OpenWrite(rdoattachment.Filename)))
{
bw.Write(rdoattachment.Bytes);
}
rdoattachment.Dispose();
}
msg.Dispose();
}
rdopststore.Dispose();


On Mon, Oct 17, 2011 at 5:23 PM, Kevin Reilly <kmreilly.ny@gmail.com> wrote:
I am testing it right now will inform you of my results.


On Mon, Oct 17, 2011 at 4:31 PM, ccurrens <notifications@codeplex.com> wrote:

From: ccurrens

If you pull the latest source, all of the memory leaks should be fixed. This changeset is the most stable we've had yet!

Read the full discussion online.

To add a post to this discussion, reply to this email (pstsdknet@discussions.codeplex.com)

To start a new discussion for this project, email pstsdknet@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com



Developer
Oct 19, 2011 at 8:21 PM

I've reviewed your code and ran it in my environment, and I've experienced no handle leaks.  After monitoring the application in Process Explorer, I can see the handle's being created, and then destroyed.  I've run it on about 30 different PSTs and at no time are there any handles left open.  I should note that the behavior of the pstsdk has changed in that Dispose() doesn't actually free the handle.  The handle is freed automatically when the variable leaves scope or is collected by the GC, whichever comes first.  Therefore, if you are checking for handles when you're still in the scope of that code that you've run above, the GC can't remove the variables since they are still on the method's stack.

 

Thanks,

Christopher

Oct 21, 2011 at 3:58 PM
Thanks for the reply.
In my code where I am creating threads that opens a PST, extracts the message data and then closes the PST I also do not see any leaks. perhaps it has something to do with the GC and the main thread. Thanks for the update.
Kevin

On Wed, Oct 19, 2011 at 3:21 PM, ccurrens <notifications@codeplex.com> wrote:

From: ccurrens

I've reviewed your code and ran it in my environment, and I've experienced no handle leaks. After monitoring the application in Process Explorer, I can see the handle's being created, and then destroyed. I've run it on about 30 different PSTs and at no time are there any handles left open. I should note that the behavior of the pstsdk has changed in that Dispose() doesn't actually free the handle. The handle is freed automatically when the variable leaves scope or is collected by the GC, whichever comes first. Therefore, if you are checking for handles when you're still in the scope of that code that you've run above, the GC can't remove the variables since they are still on the method's stack.

Thanks,

Christopher

Read the full discussion online.

To add a post to this discussion, reply to this email (pstsdknet@discussions.codeplex.com)

To start a new discussion for this project, email pstsdknet@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com