fxsaber: No, unfortunately. Renat argued that the glass is permanently streamed for the whole marketwatch. But this is not an expedient (wasteful) solution for most situations.
It's the only possible option. Streaming stakes is a native and inseparable flow of ticks.
Everything in the platform is interconnected and requires global data synchronisation. And the tick stream plays a crucial role in this.
But most importantly, the sample rate calculations above are irrelevant. They are made so clumsily (measuring anything but CopyTicks time), that it is even surprising.
Didn't look at the speed calculations, I will measure myself using the method you described above.
It's about getting into a close cache for the last 4096 quotes.
Anything that gets into it takes microseconds, and anything longer requires access to history (including disk).
Therefore, my example of fetching ticks for 20 min. on USDCHF, which is rarely updated, was in cache, but on GBPUSD, it already exceeded last 4096 ticks and had to go to the far database.
It's about getting into a close cache for the last 4096 quotes.
Anything that gets into it takes microseconds, and anything more requires access to history (including disk).
On your demo this is indeed the case. On BCS it doesn't.
Network 'xxx': authorized on BCS-MetaTrader5 through Access Server #2 (ping: 46.66 ms)
2016.10.1815:12:32.949 Test14 (Si-12.16,M1) Time: 29089 msc for1503 records 2016.10.1815:12:32.822 Test14 (Si-12.16,M1) Time: 33207 msc for1501 records 2016.10.1815:12:32.639 Test14 (Si-12.16,M1) Time: 21389 msc for1500 records 2016.10.1815:12:31.959 Test14 (Si-12.16,M1) Time: 21926 msc for1500 records
And on Alpari it's not good at all.
Network 'xxx': authorized on Alpari-MT5 through mt5.nl.3 (ping: 61.87 ms)
2016.10.1815:14:47.159 Test14 (GBPUSD,M1) Time: 31086 msc for1836 records 2016.10.1815:14:46.999 Test14 (GBPUSD,M1) Time: 30698 msc for1836 records 2016.10.1815:14:46.779 Test14 (GBPUSD,M1) Time: 46306 msc for1836 records 2016.10.1815:14:46.612 Test14 (GBPUSD,M1) Time: 30440 msc for1836 records 2016.10.1815:14:46.532 Test14 (GBPUSD,M1) Time: 36227 msc for1836 records
Therefore, my example of fetching ticks for 20 minutes on rarely updated USDCHF fits in the cache, but for GBPUSD it already exceeded the last 4096 ticks and forces me to go to a distant database.
Commented above about the inconvenience of copytix. The presented indicator is slowing down due to the fact that it is forced to call copyix several times. And all the lags are because of it. Here is the reason
When non-zero From inside CopyTicks is looking for (probably a binary search) a place to start - an index in the internal base.
Please add a function that returns the index in the database by time. Then this task will be possible to solve optimally
Now, to download ticks between dates, we must make requests for UNKNOWN number of ticks, starting from start date. And then check each time, whether it has reached the end date. And taking into account the fact, that each request copytix is very expensive, and you get such brakes.
It would be good to have a cache of the copied data requested earlier. Well and if copying is done in several steps (from[i] to from[i+ 1]), it would be logical to cache a lot after the first request.
All in all, it's not clear how the function is arranged internally. But it is hard to work with it - 100%.
long LastTime = 0; // time_msc-время последнего тика (самого свежего), полученного из истории int Count = 0; // Количество тиков в последенем запросе, у которых time_msc == LastTime
// Возвращает свежие тики, пришедшие после предыдущего вызова int GetFreshTicks( MqlTick &Ticks[], constuint flags = COPY_TICKS_INFO, constuint count = 2000 ) { int Res = 0;
if ((NewAmount > 0) && (Count < NewAmount)) { Res = ArrayCopy(Ticks, NewTicks, 0, Count);
// Взяли крайнее время из текущей истории LastTime = NewTicks[NewAmount - 1].time_msc; Count = 1;
// Находим (Count) в текущей истории количество тиков со временем LastTime for (int i = NewAmount - 2; i >= 0; i--) { if (NewTicks[i].time_msc < LastTime) break;
// Взяли свеженькие тики Amount = GetFreshTicks(Ticks); } }
Result on Metaquotes-demo
2016.10.1816:06:18.744 Test15 (GBPUSD,M1) i = 40 Amount = 1453 Sum = 79414GetMicrosecondCount()-StartTime = 584393 2016.10.1816:06:18.744 Test15 (GBPUSD,M1) i = 39 Amount = 1999 Sum = 77961GetMicrosecondCount()-StartTime = 584314 2016.10.1816:06:18.729 Test15 (GBPUSD,M1) i = 38 Amount = 1999 Sum = 75962GetMicrosecondCount()-StartTime = 568509 2016.10.1816:06:18.714 Test15 (GBPUSD,M1) i = 37 Amount = 1999 Sum = 73963GetMicrosecondCount()-StartTime = 552582 2016.10.1816:06:18.696 Test15 (GBPUSD,M1) i = 36 Amount = 1999 Sum = 71964GetMicrosecondCount()-StartTime = 536790 2016.10.1816:06:18.681 Test15 (GBPUSD,M1) i = 35 Amount = 1999 Sum = 69965GetMicrosecondCount()-StartTime = 520432 2016.10.1816:06:18.666 Test15 (GBPUSD,M1) i = 34 Amount = 1999 Sum = 67966GetMicrosecondCount()-StartTime = 504725 2016.10.1816:06:18.649 Test15 (GBPUSD,M1) i = 33 Amount = 1999 Sum = 65967GetMicrosecondCount()-StartTime = 488911 2016.10.1816:06:18.634 Test15 (GBPUSD,M1) i = 32 Amount = 1999 Sum = 63968GetMicrosecondCount()-StartTime = 473372 2016.10.1816:06:18.619 Test15 (GBPUSD,M1) i = 31 Amount = 1999 Sum = 61969GetMicrosecondCount()-StartTime = 458022 2016.10.1816:06:18.604 Test15 (GBPUSD,M1) i = 30 Amount = 1999 Sum = 59970GetMicrosecondCount()-StartTime = 442557 2016.10.1816:06:18.589 Test15 (GBPUSD,M1) i = 29 Amount = 1999 Sum = 57971GetMicrosecondCount()-StartTime = 427044 2016.10.1816:06:18.571 Test15 (GBPUSD,M1) i = 28 Amount = 1999 Sum = 55972GetMicrosecondCount()-StartTime = 411524 2016.10.1816:06:18.556 Test15 (GBPUSD,M1) i = 27 Amount = 1999 Sum = 53973GetMicrosecondCount()-StartTime = 396539 2016.10.1816:06:18.541 Test15 (GBPUSD,M1) i = 26 Amount = 1999 Sum = 51974GetMicrosecondCount()-StartTime = 381185 2016.10.1816:06:18.526 Test15 (GBPUSD,M1) i = 25 Amount = 1999 Sum = 49975GetMicrosecondCount()-StartTime = 366146 2016.10.1816:06:18.511 Test15 (GBPUSD,M1) i = 24 Amount = 1999 Sum = 47976GetMicrosecondCount()-StartTime = 351066 2016.10.1816:06:18.496 Test15 (GBPUSD,M1) i = 23 Amount = 1999 Sum = 45977GetMicrosecondCount()-StartTime = 336183 2016.10.1816:06:18.481 Test15 (GBPUSD,M1) i = 22 Amount = 1999 Sum = 43978GetMicrosecondCount()-StartTime = 321109 2016.10.1816:06:18.466 Test15 (GBPUSD,M1) i = 21 Amount = 1999 Sum = 41979GetMicrosecondCount()-StartTime = 306119 2016.10.1816:06:18.449 Test15 (GBPUSD,M1) i = 20 Amount = 1999 Sum = 39980GetMicrosecondCount()-StartTime = 288558 2016.10.1816:06:18.434 Test15 (GBPUSD,M1) i = 19 Amount = 1999 Sum = 37981GetMicrosecondCount()-StartTime = 273758 2016.10.1816:06:18.419 Test15 (GBPUSD,M1) i = 18 Amount = 1999 Sum = 35982GetMicrosecondCount()-StartTime = 259255 2016.10.1816:06:18.406 Test15 (GBPUSD,M1) i = 17 Amount = 1999 Sum = 33983GetMicrosecondCount()-StartTime = 244750 2016.10.1816:06:18.391 Test15 (GBPUSD,M1) i = 16 Amount = 1999 Sum = 31984GetMicrosecondCount()-StartTime = 230100 2016.10.1816:06:18.371 Test15 (GBPUSD,M1) i = 15 Amount = 1999 Sum = 29985GetMicrosecondCount()-StartTime = 216143 2016.10.1816:06:18.361 Test15 (GBPUSD,M1) i = 14 Amount = 1999 Sum = 27986GetMicrosecondCount()-StartTime = 201830 2016.10.1816:06:18.346 Test15 (GBPUSD,M1) i = 13 Amount = 1999 Sum = 25987GetMicrosecondCount()-StartTime = 186887 2016.10.1816:06:18.331 Test15 (GBPUSD,M1) i = 12 Amount = 1999 Sum = 23988GetMicrosecondCount()-StartTime = 172667 2016.10.1816:06:18.311 Test15 (GBPUSD,M1) i = 11 Amount = 1999 Sum = 21989GetMicrosecondCount()-StartTime = 158356 2016.10.1816:06:18.299 Test15 (GBPUSD,M1) i = 10 Amount = 1999 Sum = 19990GetMicrosecondCount()-StartTime = 143450 2016.10.1816:06:18.289 Test15 (GBPUSD,M1) i = 9 Amount = 1999 Sum = 17991GetMicrosecondCount()-StartTime = 128520 2016.10.1816:06:18.269 Test15 (GBPUSD,M1) i = 8 Amount = 1999 Sum = 15992GetMicrosecondCount()-StartTime = 114209 2016.10.1816:06:18.256 Test15 (GBPUSD,M1) i = 7 Amount = 1999 Sum = 13993GetMicrosecondCount()-StartTime = 100016 2016.10.1816:06:18.246 Test15 (GBPUSD,M1) i = 6 Amount = 1999 Sum = 11994GetMicrosecondCount()-StartTime = 85745 2016.10.1816:06:18.231 Test15 (GBPUSD,M1) i = 5 Amount = 1999 Sum = 9995GetMicrosecondCount()-StartTime = 71438 2016.10.1816:06:18.219 Test15 (GBPUSD,M1) i = 4 Amount = 1999 Sum = 7996GetMicrosecondCount()-StartTime = 57293 2016.10.1816:06:18.204 Test15 (GBPUSD,M1) i = 3 Amount = 1999 Sum = 5997GetMicrosecondCount()-StartTime = 43192 2016.10.1816:06:18.181 Test15 (GBPUSD,M1) i = 2 Amount = 1999 Sum = 3998GetMicrosecondCount()-StartTime = 28943 2016.10.1816:06:18.171 Test15 (GBPUSD,M1) i = 1 Amount = 1999 Sum = 1999GetMicrosecondCount()-StartTime = 15160
80K ticks in half a second - slow! Saying that we should have requested more ticks instead of 2000 ticks at a time is of course possible. What is the optimal request value then?
fxsaber: 80K ticks in half a second is slow! Saying that it was necessary to request not 2000 ticks at a time, but more, of course, is possible. What request value is optimal then?
It's optimal to download what you need to deep down to yourself once, and then only download new ones from the nearby cache in microseconds.
If every time you make deep queries with falling down to the disk, then of course it's your own fault.
No, unfortunately. Renat argued that the glass is permanently streamed for the whole marketwatch. But this is not an expedient (wasteful) solution for most situations.
It's the only possible option. Streaming stakes is a native and inseparable flow of ticks.
Everything in the platform is interconnected and requires global data synchronisation. And the tick stream plays a crucial role in this.
This is the only possible option. Streaming rates is a native and inseparable flow of ticks.
Everything in the platform is interconnected and requires global data synchronisation. And the tick stream plays a critical role in this.
Well, what if no EA is subscribed to BookEvent?
Do you recommend cutting MarketWatch to improve performance?
But most importantly, the sample rate calculations above are irrelevant. They are made so clumsily (measuring anything but CopyTicks time), that it is even surprising.
Well, what if no EA is subscribed to BookEvent?
Do you recommend cutting Marketwatch to improve performance?
Cut it if you want.
That's how you test CopyTicks:
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
void OnTick()
{
ulong from =(TimeTradeServer()-1200)*1000;
ulong ticks =GetMicrosecondCount();
int records=CopyTicks(_Symbol,ExtArr,COPY_TICKS_INFO,from,2048);
ticks=GetMicrosecondCount()-ticks;
Print("Time: ",ticks," msc for ",records," records");
}
Here is the output in microseconds: 95 microseconds per sample of 2048 INFO ticks for the last 20 minutes
I ate my cat's eye out on this already. I've done a lot of tests with CopyTicks. This time I have decided that it is enough to run the indicator.
Your variant.
2016.10.18 14:29:11.966 Test14 (GBPUSD,M1) Time: 20263 msc for 2048 records
2016.10.18 14:29:10.841 Test14 (GBPUSD,M1) Time: 13190 msc for 2048 records
2016.10.18 14:29:07.788 Test14 (GBPUSD,M1) Time: 13344 msc for 2048 records
2016.10.18 14:29:07.738 Test14 (GBPUSD,M1) Time: 12751 msc for 2048 records
It's about getting into a close cache for the last 4096 quotes.
Anything that gets into it takes microseconds, and anything longer requires access to history (including disk).
Therefore, my example of fetching ticks for 20 min. on USDCHF, which is rarely updated, was in cache, but on GBPUSD, it already exceeded last 4096 ticks and had to go to the far database.
If I set ulong from =(TimeTradeServer()-600)*1000, then it will fit on GBPUSD as well.
It's about getting into a close cache for the last 4096 quotes.
Anything that gets into it takes microseconds, and anything more requires access to history (including disk).
On your demo this is indeed the case. On BCS it doesn't.
2016.10.18 15:12:32.949 Test14 (Si-12.16,M1) Time: 29089 msc for 1503 records
2016.10.18 15:12:32.822 Test14 (Si-12.16,M1) Time: 33207 msc for 1501 records
2016.10.18 15:12:32.639 Test14 (Si-12.16,M1) Time: 21389 msc for 1500 records
2016.10.18 15:12:31.959 Test14 (Si-12.16,M1) Time: 21926 msc for 1500 records
And on Alpari it's not good at all.
2016.10.18 15:14:47.159 Test14 (GBPUSD,M1) Time: 31086 msc for 1836 records
2016.10.18 15:14:46.999 Test14 (GBPUSD,M1) Time: 30698 msc for 1836 records
2016.10.18 15:14:46.779 Test14 (GBPUSD,M1) Time: 46306 msc for 1836 records
2016.10.18 15:14:46.612 Test14 (GBPUSD,M1) Time: 30440 msc for 1836 records
2016.10.18 15:14:46.532 Test14 (GBPUSD,M1) Time: 36227 msc for 1836 records
Therefore, my example of fetching ticks for 20 minutes on rarely updated USDCHF fits in the cache, but for GBPUSD it already exceeded the last 4096 ticks and forces me to go to a distant database.
Commented above about the inconvenience of copytix. The presented indicator is slowing down due to the fact that it is forced to call copyix several times. And all the lags are because of it. Here is the reason
Forum on trading, automated trading systems and testing trading strategies
Testing 'CopyTicks'
fxsaber, 2016.10.11 19:23
What is the optimal (fastest) algorithm to get the ticks from_time to_time?A solution has been proposed
Forum on trading, automated trading systems and trading strategy testing
Testing 'CopyTicks'
fxsaber, 2016.10.12 08:44
When non-zero From inside CopyTicks is looking for (probably a binary search) a place to start - an index in the internal base.
Please add a function that returns the index in the database by time. Then this task will be possible to solve optimally
Now, to download ticks between dates, we must make requests for UNKNOWN number of ticks, starting from start date. And then check each time, whether it has reached the end date. And taking into account the fact, that each request copytix is very expensive, and you get such brakes.
And on Alpari it's not good at all
Alpari has been fixed, it seems. BCS has not.
It would be good to have a cache of the copied data requested earlier. Well and if copying is done in several steps (from[i] to from[i+ 1]), it would be logical to cache a lot after the first request.
All in all, it's not clear how the function is arranged internally. But it is hard to work with it - 100%.
int Count = 0; // Количество тиков в последенем запросе, у которых time_msc == LastTime
// Возвращает свежие тики, пришедшие после предыдущего вызова
int GetFreshTicks( MqlTick &Ticks[], const uint flags = COPY_TICKS_INFO, const uint count = 2000 )
{
int Res = 0;
MqlTick NewTicks[];
const int NewAmount = CopyTicks(Symbol(), NewTicks, flags, LastTime, count);
if ((NewAmount > 0) && (Count < NewAmount))
{
Res = ArrayCopy(Ticks, NewTicks, 0, Count);
// Взяли крайнее время из текущей истории
LastTime = NewTicks[NewAmount - 1].time_msc;
Count = 1;
// Находим (Count) в текущей истории количество тиков со временем LastTime
for (int i = NewAmount - 2; i >= 0; i--)
{
if (NewTicks[i].time_msc < LastTime)
break;
Count++;
}
}
return(Res);
}
#define TOSTRING(A) " " + #A + " = " + (string)(A)
void OnTick( void )
{
// возьмем тики с начала утренней сессии
LastTime = (TimeCurrent() - (TimeCurrent() % (24 * 3600))) * 1000;
MqlTick Ticks[];
int i = 0;
int Sum = 0;
const ulong StartTime =GetMicrosecondCount();
// Взяли свеженькие тики
int Amount = GetFreshTicks(Ticks);
while (Amount > 0)
{
Sum += Amount;
i++;
Print(TOSTRING(i) + TOSTRING(Amount) + TOSTRING(Sum) + TOSTRING(GetMicrosecondCount() - StartTime));
// Взяли свеженькие тики
Amount = GetFreshTicks(Ticks);
}
}
2016.10.18 16:06:18.744 Test15 (GBPUSD,M1) i = 39 Amount = 1999 Sum = 77961 GetMicrosecondCount()-StartTime = 584314
2016.10.18 16:06:18.729 Test15 (GBPUSD,M1) i = 38 Amount = 1999 Sum = 75962 GetMicrosecondCount()-StartTime = 568509
2016.10.18 16:06:18.714 Test15 (GBPUSD,M1) i = 37 Amount = 1999 Sum = 73963 GetMicrosecondCount()-StartTime = 552582
2016.10.18 16:06:18.696 Test15 (GBPUSD,M1) i = 36 Amount = 1999 Sum = 71964 GetMicrosecondCount()-StartTime = 536790
2016.10.18 16:06:18.681 Test15 (GBPUSD,M1) i = 35 Amount = 1999 Sum = 69965 GetMicrosecondCount()-StartTime = 520432
2016.10.18 16:06:18.666 Test15 (GBPUSD,M1) i = 34 Amount = 1999 Sum = 67966 GetMicrosecondCount()-StartTime = 504725
2016.10.18 16:06:18.649 Test15 (GBPUSD,M1) i = 33 Amount = 1999 Sum = 65967 GetMicrosecondCount()-StartTime = 488911
2016.10.18 16:06:18.634 Test15 (GBPUSD,M1) i = 32 Amount = 1999 Sum = 63968 GetMicrosecondCount()-StartTime = 473372
2016.10.18 16:06:18.619 Test15 (GBPUSD,M1) i = 31 Amount = 1999 Sum = 61969 GetMicrosecondCount()-StartTime = 458022
2016.10.18 16:06:18.604 Test15 (GBPUSD,M1) i = 30 Amount = 1999 Sum = 59970 GetMicrosecondCount()-StartTime = 442557
2016.10.18 16:06:18.589 Test15 (GBPUSD,M1) i = 29 Amount = 1999 Sum = 57971 GetMicrosecondCount()-StartTime = 427044
2016.10.18 16:06:18.571 Test15 (GBPUSD,M1) i = 28 Amount = 1999 Sum = 55972 GetMicrosecondCount()-StartTime = 411524
2016.10.18 16:06:18.556 Test15 (GBPUSD,M1) i = 27 Amount = 1999 Sum = 53973 GetMicrosecondCount()-StartTime = 396539
2016.10.18 16:06:18.541 Test15 (GBPUSD,M1) i = 26 Amount = 1999 Sum = 51974 GetMicrosecondCount()-StartTime = 381185
2016.10.18 16:06:18.526 Test15 (GBPUSD,M1) i = 25 Amount = 1999 Sum = 49975 GetMicrosecondCount()-StartTime = 366146
2016.10.18 16:06:18.511 Test15 (GBPUSD,M1) i = 24 Amount = 1999 Sum = 47976 GetMicrosecondCount()-StartTime = 351066
2016.10.18 16:06:18.496 Test15 (GBPUSD,M1) i = 23 Amount = 1999 Sum = 45977 GetMicrosecondCount()-StartTime = 336183
2016.10.18 16:06:18.481 Test15 (GBPUSD,M1) i = 22 Amount = 1999 Sum = 43978 GetMicrosecondCount()-StartTime = 321109
2016.10.18 16:06:18.466 Test15 (GBPUSD,M1) i = 21 Amount = 1999 Sum = 41979 GetMicrosecondCount()-StartTime = 306119
2016.10.18 16:06:18.449 Test15 (GBPUSD,M1) i = 20 Amount = 1999 Sum = 39980 GetMicrosecondCount()-StartTime = 288558
2016.10.18 16:06:18.434 Test15 (GBPUSD,M1) i = 19 Amount = 1999 Sum = 37981 GetMicrosecondCount()-StartTime = 273758
2016.10.18 16:06:18.419 Test15 (GBPUSD,M1) i = 18 Amount = 1999 Sum = 35982 GetMicrosecondCount()-StartTime = 259255
2016.10.18 16:06:18.406 Test15 (GBPUSD,M1) i = 17 Amount = 1999 Sum = 33983 GetMicrosecondCount()-StartTime = 244750
2016.10.18 16:06:18.391 Test15 (GBPUSD,M1) i = 16 Amount = 1999 Sum = 31984 GetMicrosecondCount()-StartTime = 230100
2016.10.18 16:06:18.371 Test15 (GBPUSD,M1) i = 15 Amount = 1999 Sum = 29985 GetMicrosecondCount()-StartTime = 216143
2016.10.18 16:06:18.361 Test15 (GBPUSD,M1) i = 14 Amount = 1999 Sum = 27986 GetMicrosecondCount()-StartTime = 201830
2016.10.18 16:06:18.346 Test15 (GBPUSD,M1) i = 13 Amount = 1999 Sum = 25987 GetMicrosecondCount()-StartTime = 186887
2016.10.18 16:06:18.331 Test15 (GBPUSD,M1) i = 12 Amount = 1999 Sum = 23988 GetMicrosecondCount()-StartTime = 172667
2016.10.18 16:06:18.311 Test15 (GBPUSD,M1) i = 11 Amount = 1999 Sum = 21989 GetMicrosecondCount()-StartTime = 158356
2016.10.18 16:06:18.299 Test15 (GBPUSD,M1) i = 10 Amount = 1999 Sum = 19990 GetMicrosecondCount()-StartTime = 143450
2016.10.18 16:06:18.289 Test15 (GBPUSD,M1) i = 9 Amount = 1999 Sum = 17991 GetMicrosecondCount()-StartTime = 128520
2016.10.18 16:06:18.269 Test15 (GBPUSD,M1) i = 8 Amount = 1999 Sum = 15992 GetMicrosecondCount()-StartTime = 114209
2016.10.18 16:06:18.256 Test15 (GBPUSD,M1) i = 7 Amount = 1999 Sum = 13993 GetMicrosecondCount()-StartTime = 100016
2016.10.18 16:06:18.246 Test15 (GBPUSD,M1) i = 6 Amount = 1999 Sum = 11994 GetMicrosecondCount()-StartTime = 85745
2016.10.18 16:06:18.231 Test15 (GBPUSD,M1) i = 5 Amount = 1999 Sum = 9995 GetMicrosecondCount()-StartTime = 71438
2016.10.18 16:06:18.219 Test15 (GBPUSD,M1) i = 4 Amount = 1999 Sum = 7996 GetMicrosecondCount()-StartTime = 57293
2016.10.18 16:06:18.204 Test15 (GBPUSD,M1) i = 3 Amount = 1999 Sum = 5997 GetMicrosecondCount()-StartTime = 43192
2016.10.18 16:06:18.181 Test15 (GBPUSD,M1) i = 2 Amount = 1999 Sum = 3998 GetMicrosecondCount()-StartTime = 28943
2016.10.18 16:06:18.171 Test15 (GBPUSD,M1) i = 1 Amount = 1999 Sum = 1999 GetMicrosecondCount()-StartTime = 15160
80K ticks in half a second is slow! Saying that it was necessary to request not 2000 ticks at a time, but more, of course, is possible. What request value is optimal then?
It's optimal to download what you need to deep down to yourself once, and then only download new ones from the nearby cache in microseconds.
If every time you make deep queries with falling down to the disk, then of course it's your own fault.