JS callbacks in CLR method

May 9, 2015 at 3:34 AM
Edited May 9, 2015 at 3:36 AM
Hi,

First of all let me apologise if this has asked before, I have scoured the discussions for hours now and being new to V8.NET this is a little daunting on how you operate with things on the CLR side.

I'm trying to replace Jint with V8.NET in one of my projects which had wired up a whole bunch of functions to the global object, some of them with JS callbacks, like the following example:
CLR:

void create_alias(string aliasName, string cost, int cooldown, JsValue func)
{
    ...
}
The point of this CLR method is to wire up Javascript functions to a command system in the host process which is a CLR application; when anyone executed a string command that contained aliasName in it, the function pointed to by func would execute with a set of expected function parameters.

The callback could effectively execute at any time.

The CLR method was consumed on the javascript side like this:
create_alias("test", "0", 0, function(user, parameters) {
   // callback function executes whenever user executed a "test" command.
});
I'm having a lot of trouble replicating this functionality in V8.NET. I tried changing the method signature from JSValue to Handle and passing it to Execute() but that obviously isn't going to work since that handle needs to point to a compiled script, and that is basically where I became lost.

How do I get the engine to call functions passed into CLR methods with the parameter signature? I noticed ClearScript has this functionality fairly well documented but I would rather not go down that path if I could avoid it.

Any help would be greatly appreciated. :)

Cheers,
T.
Coordinator
May 9, 2015 at 5:44 AM
V8.Net is a low level wrapper to V8, so it doesn't do any "magic" on its own. Normally a developer would have to have to use special callbacks, or create their own binding. Because of many requests, I created a feature to bind object instances - since a C# method MUST be in a class anyhow, it doesn't stand alone. I suggest instantiating your class with all required methods and set a global property to that object. Optionally, you can create a binder object that wraps this new object instead and set the properties of this binder object in the global object as well. I'm heading to bed, but I'll try to give an example tomorrow. In either case, it's literally just a few lines.
May 9, 2015 at 5:57 AM
Hi James,

Thanks for your prompt reply. Indeed, the question is not that C# methods must be inside their own classes.

To test what I want to to achieve I made my own class designed to illustrate a simple callback:
        public class test {
            public void Test(string str, JSFunction func)
            {
                func(Jist2Plugin.Environment.Engine, false, null, null);
            }
        }
where func appears to be a delegate pointing to a JS callback, although I'm no doubt misinterpreting its purpose.

On the JS side, the Test method was conumed via:
test.Test("asd", function() { dump(this) });
Which I'm sure you can guess doesn't work. Methods can live inside their own type inside V8 with SetProperty(typeof(test)), that's no issue, but I'd like to be able to enable the CLR to execute whatever function was passed into the method at a later date.

Cheers,
T.
Coordinator
May 9, 2015 at 6:39 AM
JSFunction is for function templates only. Everything is passed around using handles (values and objects, which includes function objects). Just make the parameter "func" of type InternalHandle (or even possibly ObjectHandle) and there are methods to invoke the function on the handle.
May 9, 2015 at 7:18 AM
Edited May 9, 2015 at 7:37 AM
Perfect!

After a few slaps on the forehead I now realise how easy this is. I had overlooked InternalHandle because I was thinking about it the wrong way; I had missed the fact that InternalHandle objects are created with engine.CreateValue() and not new InternalHandle(clrObject) so mechanisms like StaticCall() exist on the handle itself, rather than asking the engine object to execute a handle such like what I was so used to previously.

For the people stuck on this same problem in the future:

In a CLR method, JavaScript functions can be passed in by the InternalHandle type, and take any arbitrary number of handle parameters in which you have to create from CLR values.
public class test {
            public void Test(string str, InternalHandle func)
            {
                  InternalHandle strHandle = jsEngine.CreateValue(str);

                  func.StaticCall(strHandle);
            }
}
Which may be consumed on the JS side like so:
test.Test("test string", function(str) { alert(str); });
And assuming you have alert in javascript wired up to log an output, the mechanism works perfect.

Cheers!
Coordinator
May 9, 2015 at 1:52 PM
Glad it worked out - but why not use InternalHandle for the string also? :). You can save a step.
Coordinator
May 9, 2015 at 1:55 PM
Also, make sure to read the documentation part for handles to know how to use them properly.
May 9, 2015 at 2:10 PM
Yeah I believe I have got the usage of handles - A few times where I have dispatched a Handle for deferred execution later using the async Task mechanism the handle would point to undefined because it appears the GC doesn't know how to close over input parameters in the fashion I expected it to and the handle was submitted for garbage collection before the task has a chance to execute the function; handle.Clone() in a local had solved the problem: I do believe the internal refcount for the handle would be one more than it needed to be at the time the Task was submitted to the threadpool, but that's alright as long as it doesn't fall to zero prematurely.

Here is my setTimeout-esque method for deferred function execution:
        public void RunAfterAsync(int AfterMilliseconds, InternalHandle Func, params object[] args)
        {
            System.Threading.CancellationTokenSource source;
            InternalHandle _func = Func.Clone();

            lock (runAfterList) {
                source = new System.Threading.CancellationTokenSource();
                runAfterList.Add(source);
            }

            Action runAfterFunc = async () => {
                try {
                    await Task.Delay(AfterMilliseconds, source.Token);
                } catch (TaskCanceledException) {
                    return;
                }

                if (source.Token.IsCancellationRequested == true) {
                    return;
                }

                try {
                    _func.StaticCall(args.ToInternalHandleArray(Jist2Plugin.Environment.Engine));
                } catch (TaskCanceledException) {
                }

                if (source.Token.IsCancellationRequested == false) {
                    lock (runAfterList) {
                        runAfterList.Remove(source);
                    }
                }
            };

            Task.Factory.StartNew(runAfterFunc, source.Token);
        }
It does some internal housekeeping to make sure all deferred executors are cancelable.

Seriously loving this library once you get over the daunting hump of being completely new to it. :)

Cheers,
T.
Coordinator
May 9, 2015 at 4:30 PM
Edited May 9, 2015 at 4:37 PM
Thanks for the great feedback. ;)

The reason for your issue is probably because the the InternalHandle passed in gets destroyed automatically upon return from the CLR method (there's code that does this for cleanup when the method returns). You must always get a copy of it if you need to persist it.

In addition consider using an object reference based handle instead, so the GC will dispose the handle later, unless that doesn't matter:
ObjectHandle _func = Func;
This is one instance (inside a callback function) where "=" can be use to initially set a handle with an InternalHandle typed parameter, because if not disposed upon return, it will be taken care of automatically. In this case, _func will get it's own copy, and all should be fine.
May 10, 2015 at 1:51 AM
I do believe I may have created some handle leaks somewhere. Is there a way to debug handle leaking? I'm observing the TotalHandles property of the engine simply increase over time; should I be storing ObjectHandle instances inside the CLR instead?
Coordinator
May 10, 2015 at 3:11 AM
This is an oversight I think when I was debugging some time ago. I had disabled the V8 GC idle notification that triggers the V8 GC because of an elusive bug, and forgot to re-enable it. I think the version in the download may also be affected. I'll have to post another new release now I think. To test this, simply open the included console app and type "\gctest". If it fails, I just uploaded a fix to the dev branch you can try.

There's no official way to debug handles, except to see how I do check for state changes in the console demo project. Look for "\gctest" in the "program.cs" file and you'll see how I do it.
May 10, 2015 at 4:19 AM
\gctest on the bundled app I have here (1.5.0.2 is its version stamp) claims to have succeeded. What I did notice is that pending handles for GC always sits in the few hundreds. Is this normal behaviour?

Would it be privy to hack on the Handle mechanism and add them to a managed list on alloc and remove them upon disposal so you could take a snapshot of what V8.NET has allocated? I realise this is horrible and slow but for the purpose of profiling and plugging up holes speed really doesn't matter.
May 10, 2015 at 7:13 AM
Edited May 10, 2015 at 8:32 AM
Updated to the latest dev branch, but what does this mean? Almost all of my functions don't work anymore.
Error: Objects cannot be constructed from this function.
  Line: 10  Column: 13
  Stack:
    ....
Edit: Updated with more information:

The syntax used on the JS side is this:
jist.Random(1, 10)
jist is set as such:
_engine.GlobalObject.SetProperty("jist", JistLibrary, null, true, ScriptMemberSecurity.Locked);
where JistLibrary is an instance property of this class:
    [ScriptObject("Sealed_Object", ScriptMemberSecurity.Permanent)]
    public class JistJSLibrary : IV8NativeObject {
        protected Random rnd = new Random();
  
        /// <summary>
        /// Returns a random number between the from and to values
        /// specified.
        /// </summary>
        public int Random(int from, int to)
        {
            lock (rnd) {
                return rnd.Next(from, to);
            }
        }
        }
In a function callback, when isConstructCall is true in V8Engine_Binding.cs:1425 it throws the error mentioned. I'm sorry for hassling you but I'm having a lot of difficulty understanding what I'm constructing, why isConstructCall is true on that function template, what that even means or how it gets set under what conditions.

I've pretty much copied how the SealedObject example is instantiated in the test console app but still no luck.

Removing that line (with one other null pointer dereference elsewhere) makes it work, but I think that's what has caused regressions in the handle system resulting in leaks.

What am I doing wrong?

Cheers,
T.
Coordinator
May 10, 2015 at 4:26 PM
Odd, I'll look into this further.
Coordinator
May 11, 2015 at 3:17 AM
Edited May 11, 2015 at 3:20 AM
Your problem is not here. This code is working for me. I need to see everything. Also, you don't need to implement IV8NativeObject, unless you need it (like intercepting when 'new' is called on this type from the JS side).
May 11, 2015 at 5:39 AM
If you wouldn't mind taking a look for me that would be great - the solution whilst in transition weighs about 200MB including references, so I will gist just the files in question. If you need a compilable example I can provide it though. The product will end up open source.

https://gist.github.com/tylerjwatson/779dc4587d52d2d99b89

There are a lot of random shotgun frustrated Dispose() calls an implementations inserted to try and fix the leaks, so my apologies if it's messy.

Cheers,
T.
Coordinator
May 11, 2015 at 7:04 AM
Edited May 11, 2015 at 7:05 AM
I've located a bug that I think is affecting you as well. Try the new release and see if that fixes things. I was testing in x64, but the bug exists in x86 for the 'isConstructCall' parameter for callbacks. Looks like the binding got affected by the bug as well.
Coordinator
May 11, 2015 at 7:07 AM
Edited May 11, 2015 at 7:08 AM
Oh also, looks like the "idle notification" trigger for the V8 GC was commented out because of a nasty bug some time ago due to Google code updates. I completely forgot to put it back in, so this should be fixed also.
May 11, 2015 at 7:56 AM
Edited May 11, 2015 at 8:00 AM
Cloned git just then and built a latest version of master, still no good I'm afraid re. the Objects cannot be constructed from this function problem. :(

FWIW: This host process is strictly 32-bit only.

Imgur
Coordinator
May 11, 2015 at 8:13 AM
Double check the assembly file versions and time stamps to make sure you actually do have all the latest ones. I had the exact same issue and now it's gone today.
Coordinator
May 11, 2015 at 8:21 AM
Which Windows OS is this?
May 11, 2015 at 8:32 AM
V8.Net.dll is 1.5.19.42 - I am forced to remove the static AssemblyResolve method for the proxy loader because it has a bunch of ASP.net bootstrapper and interferes with the current working directory of the host process, it defaults to V8_Net_Proxy imports anyway which resolve fine - the host process will always unfortunately be 32-bit. That wouldn't be interfering with it in any way would it?

The OS is Windows 8.1 64-bit. If you're up to it, more than happy for you to remote in and poke around, if that takes your fancy.
May 11, 2015 at 10:50 AM
Edited May 11, 2015 at 11:29 AM
All sorted. I'm still not too sure what step fixed it for sure, I thought setting the V8.Net-SharedTypes library's arch from Any CPU to x86 fixed the problem for sure, but I reverted it back to Any CPU in an effort to reproduce the problem, but it still works now. I'm very confused as to what the fault was.

Regardless, your change to ManagedJSCallback did end up fixing the script error. Now to have a look at all these leaks!

Cheers!

Edit: Unfortunately the leaks still persist. I execute the code for(i = 0; i < 1000; i++) { planteraInvasion(); } in the console, and the engine ends up with about 40k HandleProxy pointers in its array, all of which point to the same JistTSLibrary managed instance with the same Object ID.

Tracking down the cause, it seems to happen in FunctionTemplate.cs:155 in the using statement on hThis. There are subsequent allocations for each argument of the callback which appear to deallocate properly, however it seems hThis doesn't qualify for disposal: In V8NativeObject.cs:301 the object doesn't seem to qualify for disposal because IsManagedObjectWeak is false, or another condition.

I realise you might have to speculate, but would this be caused by me doing something wrong?

Thanks again once more,

T.
Coordinator
May 11, 2015 at 1:42 PM
Didn't realize you were compiling the source. It could be that some projects didn't get rebuilt properly until you made the change. V8.Net-SharedTypes is where the fix was made.

There is one place where there may be a leak: InternalHandle _func = func.Clone();, which I never see get disposed when done.
May 11, 2015 at 2:11 PM
I did notice that and fixed it, cheers but unfortunately that code isn't being run in the example.

for(int i = 0; i < 1000; i++) { jist.Random(1, 100); } results in about 2k handles toJistJSLibrary` when run from the console app.

I made a (somewhat bad attempt) at dumping the contents of the HandleProxy pointers out under this condition which can be found at https://www.dropbox.com/s/vzf7yeiizzfhqdd/handles.zip?dl=0. Take a look from lines 45k downwards, you can see they are all copies pointing to the same ObjectID.

Cheers,
T.
Coordinator
May 11, 2015 at 3:28 PM
Thanks, I'll take a closer look.
Coordinator
May 12, 2015 at 3:10 PM
I've confirmed there is a bug preventing handles from being disposed when new values are returned from callbacks on the native side (an oversight really). I'm currently working on some handle changes that I think will fix a lot of these possible leaks. I'm hoping these new changes will iron out the process and make it more reliable.
May 13, 2015 at 9:23 AM
It's good to atleast hear that I wasn't talking nonsense - I thought it might be a bit hard to find and it's good that you actually found it. I haven't got a lot of memory to play with and this generates about ~200k handles every 10 minutes. Once these are sorted we should be ready to rock.
Coordinator
May 13, 2015 at 10:58 PM
Edited May 14, 2015 at 3:09 PM
The latest dev branch commit as the new code and binaries for the new handle system. This is a fairly major overhaul of the whole system (including the native side). I've also added a "V8.NET-Console-HandleTests" project specifically for me to debug handles, but can also serve as a good example in itself, so I left it there. It's fairly stable, but needs more testing, which I'll be doing tonight.
Marked as answer by wolfje on 5/14/2015 at 2:25 AM
May 14, 2015 at 8:12 AM
Edited May 14, 2015 at 8:56 AM
Absolutely fantastic. The library is stable so far with everything I have chucked at it.

I would like to thank you very much for your perseverance with my problems. Also love the Handles_Active et al. properties, means I can dump out the handles if I find something I'm doing wrong. I thought the leaks weren't fixed but turns out I was reporting on the wrong figure; it's probably why the original total handles were deleted.

Edit: Note to self, creating 100 million abandoned CLR objects on the native side is a very, very bad idea.
May 14, 2015 at 9:10 AM
Edited May 14, 2015 at 9:24 AM
If I may ask, what makes the number of handles pending GC not go down? Do they not get disposed unless the engine runs out of memory?

Edit: It seems that CLR objects asked for get added to an abandoned queue which they are dequeued, but CanDispose seems to never be true and they are requeued for later disposal, repeat ad infinitum.

Edit again: It seems that's because the CLR is keeping hold of them. Derr. Manually invoking GC.Collect() correctly disposes of abandoned copies. An oversight on my part, my apologies, remarking the correct answer.
Coordinator
May 14, 2015 at 2:04 PM
Edited May 14, 2015 at 2:10 PM
The CLR GC, even when calling Collect under my testing, refuses to let go of all objects for some reason, but creating more objects and calling Collect (and WaitForFinalizers) successfully shows the objects later removed. Not sure why this is yet. Any object that may have a V8 handle to it (or one of the template objects which other objects still need) goes into a pending disposal process where the native V8 handle is made weak. The object will stay pending the V8 GC (native side), until it triggers a call back to release it.
Coordinator
May 14, 2015 at 4:49 PM
FYI: I've added a section about garbage collection to the documentation to help clarify the process a bit more:
https://v8dotnet.codeplex.com/documentation
May 14, 2015 at 11:35 PM
What I did notice was that the OOB disposal of abandoned objects is indeed very expensive when the abandoned queue becomes large. In my case it is easy to generate 200k-1m abandoned CLR references and it can take up to half an hour (!) to clean them all up at 100% usage of a single thread and during this time such memory pressure will halt the VM from accepting more input, at least in my case.

The issue lies in _OnNativeGCRequested() in V8NativeObject.cs:330 where the worker thread has actually requested an object to clean up. Causing a removal from the abandoned linked list is very expensive as the object being disposed probably isn't at the end or start of the collection.

To work around this I modified _doWorkStep slightly so when it pops an IV8Disposable off the abandoned queue and it is genuinely ready to be disposed (ie CanDispose == true) it simply places the object back into the disposal queue to be cleaned up in due time. In this case, the object never gets emplaced back onto the abandoned queue, which brings the benefit of the pressure on the abandoned linked list being relaxed in due time without the expensive _abandonedQueue.Remove() calls from another thread later on in the process.

This doesn't seem to cause regressions, though that may not be the right thing to do, you might have a better way. If you're interested, I most certainly can provide a patch. I'll have to do more testing to confirm that what I'm doing is the right thing.

Cheers,
T.
Coordinator
May 15, 2015 at 2:06 AM
This was only a preview dev release for testing - I'm still working on making that part better, and welcome any feedback. I'll see what I can do to help this scenario.
May 15, 2015 at 2:46 AM
Edited May 15, 2015 at 2:47 AM
Cool. I thought that method I adopted was still leaving dangling CLR object handles so I rolled it back anyway. It seems that I am still getting a lot of dangling CLR object references. See https://www.dropbox.com/s/vzf7yeiizzfhqdd/handles.zip?dl=0

The code that invoked this was on the JS side in the console:

for(i = 0; i < 10000; i++) { tshock.Utils.GetBuffName(13); }

It should be noted that GetBuffName() is an instance method on Utils which is a static member (not property) of TShock which is also static and registered with SetProperty(typeof(TShockAPI.TShock)) which will probably affect it's GC performance. I assume statics live inside GCRoot which is why the CLR object will never get finalized and the handles to them will never get released. One of the big flaws of the host application really.

Edit: GetBuffName takes a plain int and returns a string.
Coordinator
May 15, 2015 at 3:48 AM
Can you copy this into a workable project I can compile and check? I can run my own experiments, but I'd like to be sure to include your issues as well, thanks.
May 15, 2015 at 5:31 AM
Sure. The source and binaries for my entire dev environment for this can be found here: https://www.dropbox.com/s/0svo7rvmoiywb0y/Jist2-src.zip?dl=0 weighing in at 177MB.

Sources are in the Jist2 folder, and when compiled will output the file over to ServerInstance4.2/ServerPlugins. The debug action is set to run the host application but you will need to modify it's directory for the host to run.
  • Run the host application
  • Run Jist2Console.exe to interact with the JS console in the host application
  • In the console, use for(i = 0; i < 10000; i++) { tshock.Utils.GetBuffName(13); }
  • For a simple UI for tracking handle counts in real-time, type /jist debug in the host application's console window whilst playing with it.
Cheers!
Coordinator
May 15, 2015 at 7:09 PM
FYI: It turns out part of the issue for delayed GC of objects is due to 32-bit mode vs 64. In 32-bits, it seems the GC wants to hold onto some objects longer than 64-bit. When I tried the \gctest option in the console, it works in 64-bits and fails in 32. Creating more objects eventually pushes the GC to collect for 32-bit. Not sure why, but I created a post here to ask the question if interested:

http://stackoverflow.com/questions/30254539/gc-behavior-inconsistent-between-32-bit-and-64-bit-applications
Coordinator
May 15, 2015 at 8:46 PM
Ok, I've posted another dev update which removes the objects in O(1) fashion (little to no iteration). I tried creating 10000 at a time, and when the handles get close to 50,000+ (sometimes earlier), the GC kicks in. If you create objects rapidly you cannot expect the GC to keep up. In my testing, the GC kicked in around 30k-50k objects, then it kept the # from getting too high because the cache gets a dump of new proxy handles (and there wasn't much of a collection speed difference as the number got higher).
Coordinator
May 15, 2015 at 11:11 PM
I'm running some tests on your project. Your handle total keeps rising even when one is pulled from the cache. What are you using to find the total? Where is the profile code for the pop up window? No time to look yet, will try to find it later tonight or tomorrow.

Also, I'm doing something new with the assembly resolver to help your project best organize the files (like keeping everything in a "V8.NET" sub folder).
May 16, 2015 at 12:41 AM
You'll find the code for that in CProfilerWnd.cs - hopefully it's me that has it wrong. ;)
May 16, 2015 at 8:47 AM
I understand pressure on the abandoned GC if you are creating more objects than it can dispose but the thing is I'm not creating objects; I'm simply just using CLR references that should already be there.
Coordinator
May 16, 2015 at 8:57 AM
Edited May 16, 2015 at 8:58 AM
Ok, I think I have it this time, but a lot of handles will still be created until the GC decides to collect them. That's just the nature of things.

I've improved the handle GC process. The worker was being blocked by long running scripts. I've added a queue on the native side so that when the worker detects a block, it adds it to a queue for processing in parallel with the acquiring of new native handles. Once the cache is depleted, weak handles are processed twice as fast, and the V8 GC idle notification is also triggered to help collect the weak handles building up in the V8 system.

A note on your code, in CProfilerWnd.cs, use only Jist2Plugin.Environment.Engine.TotalHandles for total handles, don't subtract anything.

The latest code and binary updates are committed to the dev branch, but I'm still running tests to make sure everything is stable.
May 16, 2015 at 9:24 AM
Will merge across and see how we go. Thank you again for your perseverance.
May 16, 2015 at 12:33 PM
What I don't understand is, how something like for (i = 0; i < 10000; i++) { dump(main) } creates 4.4 million uncollected handles to the CLR object. These aren't pending GC or abandoned; they are all references to Terraria.Main. Why do so many handle copies of a reference to a CLR object that you set with SetProperty need to be created every time you access a bound type in the CLR?
Coordinator
May 16, 2015 at 3:50 PM
Not sure, I'll try that out and look into it further.

Google V8 tracks everything (arguments passed, values returned, etc) in handles. These handles have to be wrapped so they can be exposed to the managed side. As such, careful disposal is also required. That said, I agree it does seem odd in this simple case. I'll have to see what is going on when dumping properties like this, perhaps I missed something.
Coordinator
May 16, 2015 at 6:09 PM
Edited May 16, 2015 at 7:02 PM
I can see that the dump is processing over 20 million tiles (Terraria.Main.tile {Terraria.Tile[8401, 2401]}) during the process - I'm pretty sure that's your issue here. ;) I'm looking into it a bit more, but note that even with a fix, the dump may stick here for a long time.

The problem is this line in v8engine.cs:
public InternalHandle CreateValue(IEnumerable enumerable, bool ignoreErrors); (around line 708)
The line handles[i] = CreateValue(values[i]); inside the method is in a loop initializing an array to return, and it happens to be the tile array. My guess is the dump is converting the items to an array of strings to display. The array of handles is created in one shot and sent to the server to quickly initialize a native V8 array. Bottom line, all those handles WILL be disposed eventually, but not before the array is created.

One way I can might be able to help this is to create a native array and call in to set items one by one, but that is much slower. Perhaps I'll make two versions: the current one (faster), and a slower one that kicks in if there are a large number of items to create. It would help the handles, but then everything may just look stuck from the shear number of items.

Keep in mind, in either case, you still will have 20 million BINDER objects (one per tile) referenced in the native array returned for dumping.
May 17, 2015 at 1:39 AM
Haha yeah those data structures are completely ridiculous, I agree there. You would never run this code and I understand that the handles need to be created at some point but they should be released at the return of the dump function. I can and will run the process completely out of RAM and they never get disposed at all, I have a nice 600MB handles.txt file made entirely of Terraria.Main references.

If I add hooking inside internal events and they don't clean up after themselves 100% then the process will inevitably run out of memory. It seems like everything gets taken care of apart from the property used as this.
Coordinator
May 17, 2015 at 2:44 AM
Edited May 17, 2015 at 2:49 AM
Try the new update in Dev. I don't think the handles were being disposed properly after all for arrays. I've fixed a bug there. Also, the handles shouldn't keep climbing as fast (they recycle a bit more efficiently now).

Thanks for your patience in all this. It's really helpful to have a "real world" scenario to work with. ;)
May 17, 2015 at 3:21 AM
My pleasure, it's probably me that should be worried about being annoying. ;) The bad news is though unless I failed at merging the problem still persists:

Imgur
Coordinator
May 17, 2015 at 3:57 AM
Edited May 17, 2015 at 4:02 AM
Ok, I've posted more changes to dev.

Side note: In "JSEnvironment.cs", add this first line:
V8Engine.AlternateRootSubPath = "ServerPlugins"; <= ADD THIS
this._engine = new V8Engine();
this._plugin = plugin;
Then put all libraries in "ServerPlugins".
I created a PRE-build event to do this for Jist2Plugin:
xcopy "...location to source...\V8.NET\Source\bin\$(ConfigurationName)\*.*" "$(OutDir)" /S /Y /D
For the issue above, try hiding the property by setting 'ScriptMemberSecurity.Hidden' on it like so:
var typeBinder = _engine.RegisterType<Terraria.Main>("main", true, ScriptMemberSecurity.Permanent);
typeBinder.ChangeMemberSecurity("tile", ScriptMemberSecurity.Hidden);
or even just hide all enumerable types this:
var typeBinder = _engine.RegisterType<Terraria.Main>("main", true, ScriptMemberSecurity.Permanent);
foreach (var member in typeof(Terraria.Main).GetMembers(BindingFlags.Public | BindingFlags.Static | BindingFlags.FlattenHierarchy))
{
    var type = member is PropertyInfo ? ((PropertyInfo)member).PropertyType : member is FieldInfo ? ((FieldInfo)member).FieldType : null;
    if (type != null && type != typeof(string) && typeof(IEnumerable).IsAssignableFrom(type))
        typeBinder.ChangeMemberSecurity(member.Name, ScriptMemberSecurity.Hidden);
}
If hidden, you can still access it. It just gets hidden from enumeration.

I'm still going to keep poking around to see where I can improve stuff.
Coordinator
May 17, 2015 at 4:35 AM
Edited May 17, 2015 at 4:46 AM
Fixed a lock issue with updating abandoned objects. Just posted another dev update.

I cannot get the handles that high under the new changes. The GC keeps kicking in to recycle handles too often. Instead, what I keep getting is an out of memory error that shows in the jist plugin console.
Coordinator
May 17, 2015 at 4:51 AM
Call your jist.InvokeGC() (JistJSLibrary.cs) method to test and see if all the handles clean up. With the memory errors, I cannot proceed here (it works as expected until then).

Also, this method is not complete. It should contain this:
GC.Collect();
GC.WaitForPendingFinalizers();
May 17, 2015 at 5:20 AM
I've just had a family emergency come up and I will have to take a break from this for a few days. My apologies and I'll resume asap.
Coordinator
May 17, 2015 at 5:20 AM
FYI: In x86 I've noticed the managed GC keeps objects and doesn't let them go for some time (lags a lot more). I have no control over that, so it will appear like the handles are still in use.

I'll see if there are any more optimizations I can do tomorrow.
Coordinator
May 17, 2015 at 5:22 AM
So sorry to hear that; hope it works out. :/
Coordinator
May 21, 2015 at 4:19 AM
Finished the new handle system, but your project is giving me a strange error trying to load the V8.Net DLL. I think the V8.Net.dll is being loaded as a plugin, and it shouldn't be, and is causing other issues. Anyhow, I'll try again tomorrow. The new code+binaries are on dev.
May 29, 2015 at 11:02 AM
Hi again, sorry for the delay, it had ended up being quite a difficult two weeks.

I will resume and pick up the new system and see how we go.

Cheers!
May 30, 2015 at 5:49 AM
Hi again.

I can still run the process out of memory by doing something like this:
for(i = 0; i < 1000; i++) { main.player }
I realise this is a silly practise and there is a lot of work binding such a large array of stuff but I assume everything should be cleaned up by the time the iterator returns. Subsequent execution of this snippet will eventually grind the process out of memory, and a manual GC does not solve the problem. Any ideas?

Cheers!
Coordinator
Jun 2, 2015 at 7:07 PM
That's a bad assumption unfortunately. The GC decides when to clean things up. V8.Net has no control over that. It cannot track object references like the GC does (not possible).
Coordinator
Jun 2, 2015 at 7:20 PM
I just tested this again - it works fine. I just ran the code above and all unused handles ended up in the cache at the end. Perhaps your stats are wrong. I changed some of your code:

In JistRestInterace.cs, lines 32-34:
            { "TotalHandles", Jist2Plugin.Environment.Engine.TotalHandles },
            { "TotalHandlesPendingGC", Jist2Plugin.Environment.Engine.TotalHandlesPendingDisposal },
            { "TotalHandlesCached", Jist2Plugin.Environment.Engine.TotalHandlesCached }
In CProfilerWnd.cs:
        lblTotalHandles.Text = Jist2Plugin.Environment.Engine.TotalHandles.ToString();
        lblPendingGC.Text = Jist2Plugin.Environment.Engine.TotalHandlesPendingDisposal.ToString();
        ...
        ...
        lblCached.Text = Jist2Plugin.Environment.Engine.TotalHandlesCached.ToString();
Make sure you have the latest dev binaries, and make sure you delete all copies of the V8.Net assemblies to be sure (this includes the bin/debug/release folders).
Coordinator
Jun 2, 2015 at 7:22 PM
A tip: Check "V8Engine.Version" to make sure the correct (latest) version is loading.
Coordinator
Jun 2, 2015 at 7:27 PM
Edited Jun 2, 2015 at 7:30 PM
More changes:

In Jist2Plgin.cs, line 69:
    public override void Initialize()
    {
        V8.Net.Loader.AlternateRootSubPath = "ServerPlugins"; // (prefix the plugins folder)
        V8.Net.Loader.ResolveDependencies();
  • Create a new project folder "V8.NET" and put all binaries into it. Set all files to copy if newer.
  • Change your project references to reference V8.Net and V8.SharedTypes from this folder instead.
  • Set the reference properties for V8.Net and V8.Net.SharedTypes to "Copy local: false" (only V8.Net.Loader should be copied).
  • Search your server folder for any traces of V8* and delete everything.