Crazy cache of testing agents - page 4

New comment
 
Renat Fatkhullin:

Let me get this straight:

  • Although you put agents online, they hardly ever worked because they didn't get enough orders. Not even 0.1% of the time they were not working. So you can't talk about "80 agents around the clock" at all

3 out of four systems are servers, and operate 24/7. The 4th machine occasionally shuts down. I didn't start using all of these machines at the same time, but there were a dozen of weaker computers besides them, you might have noticed that. And my agents have been in the cloud for much longer than half a year. Why agents don't get tasks is up to you.


Renat Fatkhullin:
  • You can't compare runs - they're all different for different tasks. You only have to compare the equivalent power - the quanta are calculated.
  • The figures show that the network is very profitable for consumers
That's a job my farm does in 8-10 hours. And there's no way it's eating up 10 quid worth of electricity.

I don't know what big profit you are talking about, if you have to leave your machine on for months to get a 5 min speed boost.
 
Renat Fatkhullin:

Stop making a fuss.

Want to play a real showdown on tester efficiency and performance? Try to take one agent instance, one simple task, and log all the resources in single, re-pass and optimisation mode. Then I'll quickly get you back down to earth. If you don't retract your words yourself after actually evaluating the task.

And ssd is more important to you - you didn't spend so much time describing it for nothing. And you haven't even deigned to think about my explanations. You just pressed a button, all of a sudden you're wasting resources. And you do not care to assess what is really under the bonnet, what is the amount of data.

You don't understand at all %) in your tester the bottleneck is the hard disk. And the problem is that the cores are standing and waiting for data from this slow disk . And I will make you the tests - and a more advanced version.
 
alrane:
You don't understand at all %) The bottleneck in your tester is a hard disk . And I will make you tests - and a more advanced version.

You're the one who doesn't quite understand who you're talking to about technical issues.

I, unlike you, have already spent a third of my life writing metatrader testers and optimising them.

 
alrane:
That's the job my farm does in eight to ten hours. And in no way does it eat up 10 quid worth of electricity.
The farm+setup+support cost money.
 
Renat Fatkhullin:

You're the one who doesn't quite understand who you're talking to about technical issues.

I, unlike you, have already spent a third of my life writing metatrader testers and optimising them.

Yes, but for some reason you don't give an answer to the question of HOW to solve this problem. Perhaps, your religion does not allow you to get down to it.

Please, I am asking you to use sliding test on EURUSD for one year. A single test run:


Now 100 runs:


Yes, it's working great on a brisk run.

Now let's try a heavier EA on the same period:




And what do we see? The CPUs are idle, the memory is empty, one drive is running. And the crux of the question was HOW to avoid this. Yes, it is a small number of runs. If we have thousands of them, the CPUs will be more loaded in time, when the data is cached, and the disk will be mostly used for reading. But it will still be far from being 100% utilized, the cores will be waiting for the hard drive anyway, because its speed is just too low for such a number of agents.

I have actually solved this problem long ago using my own resources (look at the date of this thread). And obtained speed increase by several times (just distributing agent cache on different media). But perhaps there's a better solution, perhaps I should pay attention to something when writing the Expert Advisor.

Or have any other thoughts.

Stop aggravating already, I'm not aiming to insult you. I came here for help.
 
fxsaber:
The farm+setup+support cost money.
What kind of money? I set it up myself, the machines are mine and strategy testing is far from their main task. By farm, I mean my bundle of several machines.
 
alrane:
On my mix of 4 x computers in 80 agents - about 6-10 hours.

Let's say 10 hours, so on one PC for 40 hours - what kind of PC write?

As I wanted to calculate the real costs are interesting, if you use a PC 24/7/365 - it turns out that the hourly consumption of 0.25 cents, and for the year 0.25 * 24 * 365 = 2190 $ - for the optimization, plus the fact that we get the result is 480 times faster.

Renat, and if I agree to get 100 times faster, then you can pay 4.8 times less - then also weak machines will be in demand in the cludes.

 
First machine: Two Xeon E5-2690 3.2GHz processors (32 logical cores), 64GB RAM

Second machine: Two Xeon E5-2670 (32 cores), 32GB RAM

Two other machines 8 cores each Xeon E3-1241v3 with 32GB RAM



IMHO, unfortunately the cloud (at least in my case), is of no use. I've been looking at it long enough to see if it makes sense
 
In fact, if interested, here are the statistics (not all cars involved are displayed):

 
alrane:
Actually, if interested, here are the statistics (not all machines involved are displayed):

Processors are different - it would be good to get data with one processor - find out the optimization time at your place and in the cloud, and the cost of work in the cloud accordingly. It turns out that the cloud is very expensive - 2k for the equivalent of a PC for a year. And, I wonder how much they pay per hour for such a PC - I am curious to know the markup :)

12345678910
New comment
Reason: