Taking Neural Networks to the next level - page 18

Bayne
971
Bayne  
Bayne:

edit: Okay short thing to add: if we have ECN Brokers. Why dont we use any data from the internet with higher quality for the timespans with horrible data? (wheter it be tickstory,dukaskopy or another brokers data..)

What about this possible soultion?

NELODI
355
NELODI  
Chris70:

I guess that's up to the freedom of imagination of the person who designs the model; the most obvious choice would probably be as simple as: hidden ("unobserved") state 1="price moves up" vs. hidden state 2="price moves down" (within a defined time interval, a next renko, a next range bar...) --> translating into trading action buy or sell.

There are times when it is better to stay away from the market, instead of buying or selling, because doing either would result in a loss.

When trade direction is undecided, price either does not move at all, or there are sudden spikes in both directions. There's also a cost associated with opening a trade (spread and broker commission), so you actually need 3 possible outputs from your network: buy, sell or wait. Or, what might also work, an output with a range from -1 to 1, where you'd only sell when the output is below -0.9 and buy if the output is above +0.9, but wait in all other cases. You could pick a different range if you wanted the network to trade more often, but I guess you get the idea.

You also need a separate output to decide when it is time to CLOSE your buying or selling position(s), because you can't use a signal for opening a position to close all possitions in the opposite direction, since that would result in closing most of your trades with minimal gains or a loss (unless there is a lot of up and down movement and the market is ranging in a perfect horizontal channel, which is rarely the case).

NELODI
355
NELODI  

Bayne:

Why dont we use any data from the internet with higher quality for the timespans with horrible data? (wheter it be tickstory,dukaskopy or another brokers data..)

I know that Chris believes the problem is in the quality of data from Brokers, but I don't believe that is the case. The actual problem is that the market behaves "irrationally" quite often (on all time-frames) and you need to take that into account when making your trade decision. Even when a trade direction is obvious, there can be spikes in either direction, after which the price returns to the "expected" range, but you can't ignore these, because they aren't just noise. They actually happen. And these spikes are the reason why the majority of traders keep losing money, even when they are right about the general trade direction.
Brian Rumbles
926
Brian Rumbles  

@NELODI I like your way of thinking in terms of trading, but don't you find the spike will catch you out when it happens initially and only after that you can decide to stay out of the market? So you still will get a loss if it is not in your direction. Any ideas?


For example on EURUSD yesterday 15 oct at 17:15 broker time GMT+1 I think, the spike on 15min chart got me on demo

NELODI
355
NELODI  
For example on EURUSD yesterday 15 oct at 17:15 broker time GMT+1 I think, the spike on 15min chart got me on demo

EURUSD on Oct 15th between 17:00 and 17:15 was a strong reversal of a down-trend on the shorter time-frame, caused by a continuation of an up-trend on a higher time-frame, as the price became very attractive to buyers when it surged below the highest low of a trend moving up on the higher time-frame. If I'd looked at the 15 minute chart, I definitely wouldn't have started selling at 17:00, because the price was already at a very low spot. You always have to keep an eye on higher time-frames when making your trade decisions. I'd also advice staying away from the market when it at a very high or a very low price, because opening a trade in either direction at that point would be gambling.

But since you've mentioned it, I do see how such a sudden trend reversal can be seen as "irrational" behavior if you only look at the time-frame you are trading.

Brian Rumbles
926
Brian Rumbles  

Thanks, yes the higher timeframes are worth looking at. What do you use to model the trend and expected range?

As for whether it is irrational or not I could never say because I don't have any info, I guess if I don't like the movement I could call it irrational 

NELODI
355
NELODI  
Brian Rumbles:

Thanks, yes the higher timeframes are worth looking at. What do you use to model the trend and expected range?

As for whether it is irrational or not I could never say because I don't have any info, I guess if I don't like the movement I could call it irrational 

I open and close my trades on M1 charts, but I use indicators which show me all the price movements of the last 5 months (from M1 to MN x 5), with moving averages calculated based on 9 time-frames, starting from the hourly and up to the weekly averages. And for the detailed moves, I use high and low points of 6 trend-lines (upper, lower and average) calculated based on H1 and H4 time-frames (60 and 240 periods). You can download my indicators from the CodeBase, if you want to see the actual calculations (search for "NELODI Trading Terminal"). But then again, it's not enough to just look at the indicators. You have to actually use them and actively trade for hundreds of hours (if not thousands) to develop an instinct. In fact, it's not the indicators that do all the work, it's your Brain.

So, the actual decision making process is done all "under-the-hood" in your Bran. What you end up with, after having traded for hundreds or thousands of hours, is an instinct that tells you whether the price is at a very attractive spot, a rather cold spot or at a dangerous level. It's just like looking at a picture and thinking: "this looks nice" or "boring" or "this is really ugly". And you get there by trading, because after each trade, you will receive positive or negative feedback (your profit or loss) and that is eventually going to form an opinion when you see something similar in the future.

What I call "irrational", is when you see something happen 100 times, but then you observe the exact opposite 101st time, before it returns back to what you'd actually expect (because it happened 100 times before). And things like that are impossible to predict, because there's a lot of people today who trade like it is gambling, but are highly leveraged and have a lot of capital. That's what causes these "irrational" spikes, which are unavoidable. And if you also trade with leverage, such "irrational" moves can get you into serious trouble, where most of your trades end up being closed with a loss, either due to a margin call, or your own stop loss, before the price returns back to normal.

Because of such spikes, I've had a lot of my "good" trades closed with a loss, before the price returned back to normal and move in the direction I was anticipating. The only way to minimize such losses, is to minimize your exposure, but doing that also minimizes your potential for profit, so ... I don't know. I guess, what it boils down to, is how much risk you are willing to take when trading. Higher risk always means a higher potential for profits AND losses.

Brian Rumbles
926
Brian Rumbles  

I did look at your terminal before but I didn't see it explained how to look at it correctly 

I have practiced a bit manual trading and yes can agree that it is necessary to see the context of good conditions or bad conditions. Conditions that can be explained as you said by ranging with a trend are fine but spikes are problems.

I think the condition may stabilize but the price does not always return in the initial direction, so the cost cannot often be recovered by the initial trade, only with new profitable trades.

Obviously with this type of behavior it turns a profitable strategy into a losing strategy or breakeven if lucky.

I agree to minimize exposure and try to select conditions that suit your model, but the spike may come any time ( as far as I can tell). Therefore it is probably necessary to increase the lot size to make up for the bad conditions when the conditions are good again and reset the lot size after recovering and profiting.


I'm sure it doesn't probably interest anyone but the model I am using is MA2 shift 10 and Lin Reg 12, you will see the trend and ranging and can trade it but the model is not good for all conditions as mentioned above.

NELODI
355
NELODI  
Brian Rumbles:

I did look at your terminal before but I didn't see it explained how to look at it correctly

There is no simple explanation for using my indicators. Their job is to show you the most relevant information about historical price movements, but there is so much data in them, that the only thing capable of making a trade decision is your Brain. And the only way to train your brain to do that, is by using these indicators for hundreds or thousands of hours, until you develop an instinct for trading with them. You can probably achieve good accuracy by using any combination of indicators, even if they didn't contain as much information as you get with my indicators, but ... as Chris said when talking about Neural Networks (and a Brain is a very complex Neural Network), you will get the best results if you allow the Neural Network (in this case, your Brain) to observe as much raw information as possible. And Because it is virtually impossible to look at 9 time-frames in real-time and make a quick decision, I've developed my indicators to summarize as much data as possible from higher time-frames and show me that data in the time-frame I'm trading.

Btw ... in my experience, increasing trading volume when the conditions "look good" to compensate for your losses from the past rarely works out, because doing that increases your exposure in the opposite direction and can actually result in adding to your loss and even blowing up your account, especially when you are highly leveraged. What you should do, is accept your loss from the past and look at every new trade as if it was your first. What's lost is lost. You can't correct your past mistakes by increasing your exposure and hoping for the best.
Chris70
577
Chris70  
NELODI:
I know that Chris believes the problem is in the quality of data from Brokers, but I don't believe that is the case. The actual problem is that the market behaves "irrationally" quite often (on all time-frames) and you need to take that into account when making your trade decision. Even when a trade direction is obvious, there can be spikes in either direction, after which the price returns to the "expected" range, but you can't ignore these, because they aren't just noise. They actually happen. And these spikes are the reason why the majority of traders keep losing money, even when they are right about the general trade direction.

No, that's not what I believe. Price data quality and price movements are two separate things. "Irrational" spikes happen all the time, even with perfect data quality.

And I accept that for many (most?) spikes there is just absolutely no way to anticipate them. But without anticipation, one might at least reflect why such spikes might have happend in hindsight: in the end, they're not really random, because they're the consequence of the actions of market participants who individually had a reason for their decision, that wasn't random, although the sum of thousands of such actions by many participants often appears to be near random. Okay, so what might cause spikes?

1. very high volume transactions of a single (or a few) big player(s) --> there's just no way of knowing about this in advance (not without insider information)

2. news --> no way of knowing this based on chart analysis alone (if fundamentals are added or at least time and expected impact level for scheduled news, this might change)

3. butterfly effects --> tiny movements cause an avalanche of stop triggering, fear/greed driven action, increased algo activity... and rapidly become a self-fulfilling prophecy --> because essentially they're (1) based on cause&effect (2) these causes are all part of the chart, such spikes might indeed (theoretically) be calculated/anticipated to a limited extent, but only with highly complex models (neural networks?)

4. combinations of the above: e.g. big news lead to big player involvment, seemingly unimportant news trigger butterfly effects...

And still, many spikes just can't be explained.

___

Some of the other things that were written in the last few posts are pretty obvious. I guess none of us who are part of this discussion would question that it's important to involve different time frames and that staying flat is sometimes the better option.

Please don't overinterprete what I wrote about application of Hidden Markov Models or their choice for the hidden state; they were just given as an example on a sidenote.

@Bayne: why not using other data sources? Sorry I didn't answer that yet. Because this wasn't such a big issue for me. My broker's data quality is good for the past few years, which is part of the reason why I don't train on 40 year old data. If on the other hand I train on more recent data, then why should I change my data source? I really don't NEED this very old history if for example over only five years there are about 120.000 M15 candles, which is more than enough to train on and - more importantly -  I don't even WANT these old data, because training on old data might lead to outdated trading decisions. The markets change and I want my decisions to be based more on the recent market situation then on data as old as myself ;-)