11/15/2015
Strategy Testing - Violating These Steps Will Damage Your Account - System Trader Success
Sunday, November 15th
Write For Us
About Us
Advertise
Learn to use two timeframes using EasyLanguage! Download Now...
search...
Go
Strategies
System Development
Indicators
Coding Lab
Our Experts
Contact
Strategy Testing – Violating These Steps Will
Damage Your Account
December 30, 2013 5:00 am0 commentsViews: 2360

Share
1

Tweet

Mail

Share
One of the most rewarding experiences for a TradeStation trader is to pick up a performance report that
proves their great strategy idea is indeed a profitable strategy. Strategy testing done properly, as is
outlined in this article, can verify the efficacy of your trading strategy and give you confidence to start
trading it. But be forewarned, strategy testing done improperly can lead you toward financial destruction.
http://systemtradersuccess.com/strategy-testing-violating-these-steps-will-damage-your-account/
1/9
11/15/2015
Strategy Testing - Violating These Steps Will Damage Your Account - System Trader Success
Strategy testing done incorrectly can result in false hope in a losing strategy. A trader recently shared his
experience of getting great results from strategy testing his idea, but after trading it live in the market, he
was losing money every day. He was baffled about what he did wrong. Having gotten excellent results on
his back-testing performance report, he wondered why his promising strategy was draining his trading
account. The problem was he violated several of the proper steps necessary for reliable strategy testing.
With the knowledge of how to get an accurate performance report you will be able to trust your strategy
in live trading and protect your trading account. In order to properly test a strategy, there are 5 main steps
that are vital to follow; configure TradeStation, “in-sample data” testing, “out-of-sample data” testing,
live forward testing on the simulator account, and real live trading execution.
Step 1: Configure TradeStation
Before you begin testing your data, you must configure TradeStation so that the data it pulls onto your
performance report will be accurate. Follow these 3 critical items to properly configure TradeStation.
(a) In your TradeStation platform menu, go to “format symbol” and give TradeStation a starting and
ending date to test. This historical date range is called the “in-sample data.” Do not include the most
recent six months in this “in-sample data.” The most recent six months is called the “out-of-sample data,”
and it will be used later during your “out-of-sample data” testing step.
(b) Next, in your TradeStation platform menu, go to “format strategy” and select “properties for all.”
Now select the “general” tab and enter the commissions and slippage (be as realistic as possible, or
estimate too high if you are not sure). If this step is skipped, then the strategy testing performance report
will be meaningless. If this is not done you might have a good looking performance report equity curve,
but as soon as you enter the commissions and slippage figures the equity curve can reverse into an
underwater equity curve.
(c) The last configuration step is under “properties for all” under the “general” tab. Look in the bottom
left section called “strategy testing resolution.” Check the “look-inside-bar back-testing” option and then
select the smallest time frame available for your chart style to make the strategy testing more closely
resemble live data. When strategy testing, TradeStation uses the open, high, low, and closing data, thus
the larger the time frame bar, the more distorted the strategy testing performance report can be. This
“look-inside bar back-testing” option will make the computer do a lot more strategy testing calculations.
This may really slow down your performance report generation, so please be patient. For an accurate
performance report you must use the “look-inside bar back-testing” option.
These configuration steps are critical to getting an accurate performance report, so be sure this is
completed precisely before continuing. Once TradeStation has been configured correctly, you can begin
testing your strategy.
Step 2: In-Sample Data Testing (also called “Back testing”)
You are now ready to start testing your strategy idea. We will begin with testing the “in-sample data” that
you set up for testing during the configuration steps. Begin with bringing up a TradeStation performance
report. Right now I have a performance report in front of me that I will refer to, but you will be looking at
your own performance report to analyze your own numbers. This is what we will be referring to in the
steps below. There are 7 sub-steps to “in-sample data” testing, as follows:
First, look at how many trades the strategy made. To reduce strategy testing errors, where error is defined
by [ error = 1 / Square Root (Number Trades In Test) ], you want at least 400 trades to reduce the margin
http://systemtradersuccess.com/strategy-testing-violating-these-steps-will-damage-your-account/
2/9
11/15/2015
Strategy Testing - Violating These Steps Will Damage Your Account - System Trader Success
for error to 5% in your strategy testing results. At 100 trades you have a 10% margin for error. The
greater the number of inputs in your strategy that you optimize, the greater the number of trades you need
to keep from over optimizing your strategy.
Also look at how many times the strategy traded on average per day. The more often a strategy trades the
more profit it can generate.
In the performance report that I am looking at, it traded 397 trades in the last 3 1/2 months, averaging 5.3
trades per day.
Second, look at the “Average Trade Amount.” It needs to be large enough that slow order fills and/or
larger than normal slippage does not kill the profitability of the strategy.
In my report the “Average Trade Amount” is $162.32. The commissions and slippage amount as defined
in the set up steps is already subtracted in this performance report.
65% of the time this strategy trades 1 contract.
35% of the time this strategy trades 3 contracts.
10% of the time this strategy trades 5 contracts.
Third, look to see if the “Profit Factor” and “Ratio Average Win-Average Loss” are both above 1.5 and
the percentage of winning trades around 45% or better
This strategy had a “Profit Factor” of 1.83.
This strategy had a “Ratio Average Win-Average Loss” of 2.28 (2.28 means breakeven is around 28%
“Percent Winning Trades”)
On this strategy the “Percent Winning Trades” was 44.58%.
Fourth, look at the trade list page and assess the profit run ups and draw downs column. Notice how many
trades made money and how much money they made before the trade exit occurred. Looking at what
amount of money was made in relation to the profit run up and draw down, you want to know if
managing the trades could generate more profits. The example used here shows that a good percentage of
trades made much higher profits than where the automated exit points occurred.
Fifth, look at the three draw down (DD) numbers. I like to see the largest number at 15% or less of the
“Total Net Profit” and the “Max DD” at 5% or less of the “Total Net Profit” (these numbers tell about the
draw down risk level during your trades).
Total Profit – $64,440
Peak to Valley DD – $8,960 is 13% of Total Profit
Close to Close DD – $7,120 is 11% of Total Profit
Max DD – $3,420 – 5% of Total Profit
Sixth, Looking at the “Largest Losing Trade” on the report, I like to see 5% or less of the “Total Net
Profit.” In my report the “Largest Losing Trade” that occurred was $2,580 which is 4% of “Total Net
Profit.”
Seventh, I review the length of time in the average trade. Does the average time in a trade comply with
the golden rule of trading; “cut your losses quickly and let your profits run?” You will also want to see if
the strategy is built using only profit exits (no real stop loss exits). It might have a nice looking report, but
it could show a messed up ratio between average bars per winning trade verses average bars per losing
trade if there are no stop loss exits. Here are my average bars:
http://systemtradersuccess.com/strategy-testing-violating-these-steps-will-damage-your-account/
3/9
11/15/2015
Strategy Testing - Violating These Steps Will Damage Your Account - System Trader Success
Average bars per winning trade 7.24
Average bars per losing trade 3.51 bars
This strategy complies with the golden rule of trading. Notice how it cuts losses quickly, at an average of
3.51 bars, and lets the profits run for an average of 7.24 bars.
So what does this all mean? It means this strategy has passed the historical strategy testing phase of
strategy testing.
Step 3: Out-of-Sample Data (also called “Walk Forward Testing”)
Once you have tested your “in-sample data” and have determined that your strategy is worthy of
continued testing, you can now test your strategy against the “out-of-sample data.” If you have not yet
tested your “in-sample data, do that before proceeding.
To test the “out-of-sample data” we use the most recent 6 months of data available that you reserved in
step 1(a). In step 1(b) of this article, we talked about configuring TradeStation and covered entering
commissions and slippage and using the “look-inside-bar back-testing” option, which must be used to run
any performance report used in your strategy testing. Be sure you have configured TradeStation correctly
before moving on.
Go into “format symbol” and change the date range to include ONLY the “out-of-sample data” date range
that was NOT used during the strategy testing on “in-sample data.” This is referred to as testing on the
“out-of-sample” data.
Begin with bringing up a TradeStation performance report on the “out-of-sample” data and review all the
items that we discussed in step 2 above on this “out-of-sample” performance report. The closer it
performs to the Step 2 “in-sample data” performance report, the more robust the strategy is. This suggests
that the results were not from curve fitting and you have a good chance of having a viable strategy. This
“out-of-sample” date range test is much more important than the strategy testing step on the “in-sampledata” for finding a successful strategy. It is a good idea to test multiple different “out-of-sample” date
ranges, which is called “Walk Forward Analysis.”
Robustness: Perry J. Kaufman stated, “Practically speaking, a robust trading strategy is one that produces
consistently good results across a broad set of parameter (input) values applied to many different markets
tested for many years.”
If the strategy fails during this “out-of-sample data” test, do NOT optimize using your reserved “out-ofsample data.” This would defeat this vitally important step in strategy development. You can go back to
your strategy and fix it, or else drop it and develop a new strategy idea.
One caveat – if your strategy is capitalizing on a certain market condition, like the current volatility, and
then you “out-of-sample data” test a non-volatile date range, it may not perform well, However in our
next phase of testing, “ Live Forward Testing,” it could prove to be successful since we are still in a
volatile market. You must understand why your strategy works, under what market conditions it performs
well, and in what market conditions it does not perform well.
Now that you have tested your “out-of-sample” data and your strategy is promising, you are ready to live
forward test your strategy on the simulator account.
At this point you have configured TradeStation so your performance report will be accurate, you have
http://systemtradersuccess.com/strategy-testing-violating-these-steps-will-damage-your-account/
4/9
11/15/2015
Strategy Testing - Violating These Steps Will Damage Your Account - System Trader Success
tested your “in-sample data” and your “out-of-sample” data and your strategy still looks great. Now you
are ready to live forward test your strategy on the simulator account.
Step 4: Live Forward Testing on the Simulator Account
During your live forward testing on the simulator, you want to verify that the live data feed entries and
exits are similar to historical entries and exits. After you have made live data trades for a day, save the
live trade list. Now reload this same chart so the strategy recalculates based on the historical for this same
day. Record the historical trade list and compare the live entries and exits to the historical entries and
exits. Are they the same or at least similar? Do you understand the differences and the impact your “Live
Data” test says about your strategy?
Only by monitoring the program daily can performance be seen under real “live” market conditions.
Continue Live Forward Testing on the simulator until you are totally comfortable that your strategy
works on live data. Real time results will often be less profitable than your historic results. The key
question is does the real time testing show that you have a profitable strategy that is worth trading?
Step 5: Real Live Trading Execution
Once you have done your due diligence and are comfortable with your strategy results on the simulator,
you are ready to trade live. Since you are trading a brand new strategy with real money, start with a
significantly reduced position sizing risk of 1/4 of 1% of your account equity at risk per your stop loss
point per trade. Continue trading with minimal risk until you have verified everything is working properly
within your new strategy during live order executions.
Once your strategy is making money in the live market, slowly over time begin to increase your position
sizing risk. Move your risk upward from 1/4 of 1% toward 1%. Continue trading at 1% risk until you
have several weeks to months of consistent trading performance. If you want to be aggressive and use
more, you can continue to increase toward the maximum 3% of your account equity at risk per trade.
If you follow the 5 steps as outlined in this article, you will now be able to confidently strategy test any
strategy idea that you have. Keep this article for reference so the next time you are inspired with a great
idea, you will be able to prove it out, protect your TradeStation trading account, and have confidence in
live trading your strategy.
Click on this link to learn how to quickly and accurately strategy test any trading idea you might have.
— by Mark David Johnson from http://www.customizedtrading.com. Mark David Johnson is a full time
TradeStation Trader and TradeStation EasyLanguage Programmer for the past 6 years. He has been
involved in programming for over 26 years and involved in trading for over 16 years.
System Trader Success Contributor
Contributing authors are active participants in the financial markets and fully
engrossed in technical or quantitative analysis. They desire to share their stories,
insights and discovers on System Trader Success and hope to make you a better
system trader. Contact us if you would like to be a contributing author and share
your message with the world.
http://systemtradersuccess.com/strategy-testing-violating-these-steps-will-damage-your-account/
5/9
11/15/2015
Strategy Testing - Violating These Steps Will Damage Your Account - System Trader Success
   
1
Tags: Automated Trading Development
Leave a Reply
Name (required)
E-Mail (will not be published) (required)
Website
Submit Comment
+ two = four
Notify me of followup comments via e-mail. You can also subscribe without commenting.
FaceBook Friends
System Trader Success
2,159 likes
Liked
Share
You like this
Interesting Videos
http://systemtradersuccess.com/strategy-testing-violating-these-steps-will-damage-your-account/
6/9
11/15/2015
Strategy Testing - Violating These Steps Will Damage Your Account - System Trader Success
Documentary: The W...
http://systemtradersuccess.com/strategy-testing-violating-these-steps-will-damage-your-account/
7/9
11/15/2015
Strategy Testing - Violating These Steps Will Damage Your Account - System Trader Success
The Wall Street Code
Effective Indicators for Trading Strategies
From Predictive Modeling to Optimization: The Next Frontier
Computer Algorithms Flying Quadcopters…Amazing!
High Frequency Trading and The New Algorithmic Ecosystem
The Impact of Algorithmic Trading
How Algorithms Shape Our World
http://systemtradersuccess.com/strategy-testing-violating-these-steps-will-damage-your-account/
8/9
11/15/2015
Strategy Testing - Violating These Steps Will Damage Your Account - System Trader Success
About Us
Futures Disclaimer
Income Disclaimer
Terms of Service
Privacy Policy
About Us
Futures Disclaimer
Income Disclaimer
Terms of Service
Privacy Policy
search:
search...
Go
© Copyright 2015 — System Trader Success. All Rights Reserved Designed by
http://systemtradersuccess.com/strategy-testing-violating-these-steps-will-damage-your-account/
9/9