Yes, dsBenchmark uses the creation of server side sessions to simulate multiple users by means of calling Perform Script on Server WITHOUT telling the calling script, a loop, to pause until the called script has concluded.
The default setup for FileMaker Server limits those Perform Script on Server sessions to 25. This limit can be set to any value up to a maximum of 500. To run dsBenchmark you should set this limit to 500.
In recent versions of FMS you need to user terminal to do this with:
fmsadmin set serverconfig scriptsessions=500
In earlier versions the limit number can be set in the Ui.
With Tokens set at 0 tokenisation does not operate.
With Tokens set to 1 or more the solution monitors the current speed of each task completion, amongst all the virtual client server side scripts, and adds a delay before the next call is made, in order to reduce the pressure on server.
This is predicated on our view, based on about 4 months of research, that ultimately more is done in the time if the task queue is slowed and the server encouraged to retreat from moving towards choking.
The Dequeue line shows the amount of delay being introduced by the Token mechanism.
You will realise that ds Benchmark is open to the community so you can see exactly how this is done, and no doubt improve on it, if you care to.
Are you opening a series of browsers / browser tabs on your local machine(s) and calling dsBenchmark from each?
To simulate WD sessions you must create one browser window or tab for each user.
This means you need to work hard to keep all the user sessions active so they don't close.
We tested this way with up to 15 users to obtain our figure of 32mb RAM being used per WD user session (on OS X)
Correct, the test is designed to automatically curtail when the server ceases to be performant, which I have set as the average completion time for each task exceeding, from memory, 3 seconds.
This is based on the idea that if the server isn't performant being able to support more users - who are going nowhere - is meaningless.
Hence the core dsBenchmark task is to establish on a consistent basis the normal max number of users that any given FMS deployment can support.
This then enables two comparative activities:
(a) adjusting the cache settings on FMS, and retesting seeking the optimum cache setting;
(b) running the same test on other servers in order to compare potential max load capacity.
Please note that this load capacity is not an absolute! Because as you will know the actual capacity of any deployment has a lot to do with the actual level of activity users, it is a comparative value, to enable comparisons to be made.
Hence, you can choose to run the less active user tests and you will see that the numbers who can be supported will rise.
If for example you judge that you have 25 very active users, 50 fairly active users and 25 relatively inactive users then set dsBenchmark to run a series of tests of 25 intense users, 50 busy and then 25 inactive, starting each set of tests in turn.
The big majority of our testing and development of this tool was on OS X servers. We then had several test users in various parts of the world running the beta on windows as well as mac. We have a strong impression that for any given physical resource OS X running Yosemite (10.10) was more efficient and faster than Windows.
Of course there is far more physical resource available on Windows servers but I have so far seen results on Windows that do not compare favourably with OS X equivalents. I cannot establish at present whether this is to do with how dsBenchmark tests or with how OS X and Windows manage resources and calls, or a combination of both.
The point is that whilst FileMaker Server has no control on how its competing calls for resource are handled the operating system does, and we observed a step change in OS X performance between FMS13 on OSX 10.9 and FMS14 on OSX 10.10.
It would be brilliant to be able to shed some light on the reasons for this difference in the future.