ibapi
ibapi copied to clipboard
hadrianl
How to request historical data multiple times?
I'm a bit confused on this one and can't seem to get a working example.
I create a historical data request for the SPY ticker, 1 day's worth of the latest 5 minute bars. For this simple example, I want to sleep for 5 minutes, then request historical data again. Right now, all it does is just show the bars pulled from the initial request, it doesn't seem to be updating at all.
To recreate this example, I have made the following modifications to the wrapper.go file:
var GlobalHistoricalDataSlice []types.TOHLCV
func GetHistoricalData() []types.TOHLCV {
return GlobalHistoricalDataSlice
}
func (w Wrapper) HistoricalData(reqID int64, bar *BarData) {
log.With(zap.Int64("reqID", reqID)).Info("<HistoricalData>",
zap.Any("bar", bar),
)
// Append the bar data to the GlobalHistoricalDataSlice. Bar data is in this format:
// {"level":"info","ts":1704830870.0692365,"caller":"ibapi2/wrapper.go:353","msg":"<HistoricalData>","reqID":1,"bar":"BarData<Date: 1704800100, Open: 472.910000, High: 472.910000, Low: 472.800000, Close: 472.800000, Volume: 5.000000, Average: 472.856000, BarCount: 4>"}
// Need to convert it to:
// type TOHLCV struct {
// DateTime time.Time
// Open float64
// High float64
// Low float64
// Close float64
// Volume float64
// }
// convert the datetime string "1704818100" format to Int64
var dateTime int64
dateTime, _ = strconv.ParseInt(bar.Date, 10, 64)
// Convert the int dateTime to time.Time for the TOHLCV struct
var dateTime2 = time.Unix(dateTime, 0)
GlobalHistoricalDataSlice = append(GlobalHistoricalDataSlice, types.TOHLCV{
DateTime: dateTime2,
Open: bar.Open,
High: bar.High,
Low: bar.Low,
Close: bar.Close,
Volume: bar.Volume,
})
}
As you can see, I created a get method that returns the latest data appended in the newly created GlobalHistoricalDataSlice. This just converts the BarData to a struct of my own creation in the types folder:
package types
import "time"
// Struct used when defining TOHLCV data
type TOHLCV struct {
DateTime time.Time
Open float64
High float64
Low float64
Close float64
Volume float64
}
Now the main script looks like this:
package main
import (
"fmt"
"time"
"go_ib/ibapi2"
)
func main() {
log := ibapi2.GetLogger().Sugar()
defer log.Sync()
wrapper := ibapi2.Wrapper{}
ic := ibapi2.NewIbClient(&wrapper)
if err := ic.Connect("127.0.0.1", 4002, 0); err != nil {
log.Panic("Connect failed:", err)
}
if err := ic.HandShake(); err != nil {
log.Panic("HandShake failed:", err)
}
// Create a contract for SPY ETF
spyContract := ibapi2.Contract{
Symbol: "SPY",
SecurityType: "STK",
Exchange: "SMART",
Currency: "USD",
}
// Set the request ID
reqID := ic.GetReqID()
// Get the contract details to ensure proper contract is created
// ic.ReqContractDetails(reqID, &spyContract)
// Request delayed market data for SPY
ic.ReqMarketDataType(3) // 3 or 4 for delayed, 1 when you have subscribed to real-time market data
ic.ReqHistoricalData(reqID, &spyContract, "", "1 D", "5 mins", "TRADES", false, 2, true, []ibapi2.TagValue{}) // time.Now().In(time.FixedZone("EST", -5*60*60)).Format("20060102 15:04:05")
ic.Run()
// Example log output:
/*
{"level":"info","ts":1704767599.3692198,"caller":"[email protected]/wrapper.go:353","msg":"<HistoricalData>","reqID":1,"bar":"BarData<Date: 1704186000, Open: 476.250000, High: 476.360000, Low: 476.000000, Close: 476.270000, Volume: 323.000000, Average: 476.301000, BarCount: 70>"}
...
{"level":"info","ts":1704767599.4046462,"caller":"[email protected]/wrapper.go:353","msg":"<HistoricalData>","reqID":1,"bar":"BarData<Date: 1704743700, Open: 473.510000, High: 474.000000, Low: 473.500000, Close: 473.760000, Volume: 10218.000000, Average: 473.790000, BarCount: 5146>"}
{"level":"info","ts":1704767599.4046462,"caller":"[email protected]/wrapper.go:359","msg":"<HistoricalDataEnd>","reqID":1,"startDate":"20240101 20:33:17","endDate":"20240108 20:33:17"}
*/
// Wait a bit to receive some data
<-time.After(time.Second * 15)
// First inspection of the historical data
fmt.Printf("First inspection of the historical data:\n")
for _, bar := range ibapi2.GetHistoricalData() {
fmt.Printf("Date: %v, Open: %v, High: %v, Low: %v, Close: %v, Volume: %v\n",
bar.DateTime,
bar.Open,
bar.High,
bar.Low,
bar.Close,
bar.Volume,
)
}
// Wait 10 minutes to allow for more 5-minute data to be generated by IB
<-time.After(time.Second * 610)
// Request more historical data for SPY. Run ic.Run() again?
//ic.ReqHistoricalData(reqID, &spyContract, "", "1 D", "5 mins", "TRADES", false, 2, true, []ibapi2.TagValue{})
//ic.Run()
// Second inspection of the historical data
fmt.Print("Second inspection of the historical data:\n")
for _, bar := range ibapi2.GetHistoricalData() {
fmt.Printf("Date: %v, Open: %v, High: %v, Low: %v, Close: %v, Volume: %v\n",
bar.DateTime,
bar.Open,
bar.High,
bar.Low,
bar.Close,
bar.Volume,
)
}
// Cancel the historical data request
ic.CancelHistoricalData(reqID)
ic.Disconnect()
}
You can see that I am employing the wait of about 10 minutes which should pull fresher up to date bar data after the first pull, however when you inspect the outputs, the bar data is the same. When you re-run the entire script, the more recent data IS pulled, however I thought there would be a better way to pull up to date data with the SAME ic instance without having to disconnect/reconnect/make a new client etc. I also have commented out a section that tries to run the ic.Run() method again, but again the output of bars pulled from the server doesn't change.
Any ideas?
Here's a more concise main code, and the example output I'm currently getting. This main code creates a new historical data request every minute for 15 minutes. The desired output should show all of today's bars, where the data should have a new bar added to it every minute. The code runs perfectly for about 3 minutes, and then I start getting the error shown below. I've narrowed it down to this part of the client.go file, however I don't know what the cause of it is. Would like some assistance getting a working code as it appears to be an ibapi issue:
package main
import (
"fmt"
"time"
"github.com/hadrianl/ibapi"
)
type TestWrapper struct {
ibapi.Wrapper
}
func (t *TestWrapper) HistoricalData(reqID int64, bar *ibapi.BarData) {
fmt.Printf("HistoricalData. ReqId: %d, BarData: %+v\n", reqID, bar)
}
func (t *TestWrapper) HistoricalDataEnd(reqID int64, startDateStr string, endDateStr string) {
fmt.Printf("HistoricalDataEnd. ReqId: %d, from %s to %s\n", reqID, startDateStr, endDateStr)
}
func (t *TestWrapper) HistoricalDataUpdate(reqID int64, bar *ibapi.BarData) {
fmt.Printf("HistoricalDataUpdate. ReqId: %d, BarData: %+v\n", reqID, bar)
}
func main() {
// Create a new instance of TestWrapper
wrapper := &TestWrapper{}
// Create a new instance of IbClient
client := ibapi.NewIbClient(wrapper)
// Connect to Interactive Brokers Gateway
if err := client.Connect("127.0.0.1", 4002, 0); err != nil {
fmt.Println("Error connecting to IB Gateway:", err)
return
}
// Handshake with the server
if err := client.HandShake(); err != nil {
fmt.Println("Handshake failed:", err)
return
}
fmt.Println("Establishing connection...")
// Wait for the connection to be established
<-time.After(time.Second * 2)
fmt.Println("Connection established!")
// Define SPY contract
spyContract := ibapi.Contract{
Symbol: "SPY",
SecurityType: "STK",
Exchange: "SMART",
Currency: "USD",
}
// Set the request ID
reqID := client.GetReqID()
client.ReqMarketDataType(3) // 3 or 4 for delayed, 1 when you have subscribed to real-time market data
// Print query for historical data in real-time for 15 minutes, every minute
startTime := time.Now()
for time.Since(startTime) < 15*time.Minute {
// Request latest historical data for today
client.ReqHistoricalData(reqID, &spyContract, "", "1 D", "1 min", "TRADES", false, 2, true, nil)
client.Run()
// Wait for historical data to be received/run again after 60 seconds
<-time.After(time.Second * 60)
// Cancel the current historical data request so we can start another one next iteration
client.CancelHistoricalData(reqID)
}
// Disconnect from IB Gateway
client.Disconnect()
fmt.Println("Disconnected.")
}
PS I:\nasty\Golang_Projects\Go_Interactive_Brokers> go run .
Establishing connection...
Connection established!
{"level":"info","ts":1708970108.8783593,"caller":"[email protected]/client.go:3048","msg":"run client"}
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708938000, Open: 507.320000, High: 507.320000, Low: 507.220000, Close: 507.220000, Volume: 9.000000, Average: 507.275000, BarCount: 3>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708938060, Open: 507.260000, High: 507.260000, Low: 507.260000, Close: 507.260000, Volume: 3.000000, Average: 507.260000, BarCount: 2>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708938120, Open: 507.320000, High: 507.320000, Low: 507.310000, Close: 507.310000, Volume: 2.000000, Average: 507.314000, BarCount: 2>
...
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708969020, Open: 507.650000, High: 507.670000, Low: 507.580000, Close: 507.590000, Volume: 319.000000, Average: 507.634000, BarCount: 211>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708969080, Open: 507.580000, High: 507.620000, Low: 507.470000, Close: 507.490000, Volume: 289.000000, Average: 507.527000, BarCount: 193>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708969140, Open: 507.480000, High: 507.520000, Low: 507.410000, Close: 507.470000, Volume: 460.000000, Average: 507.454000, BarCount: 276>
HistoricalDataEnd. ReqId: 1, from 20240225 11:55:09 to 20240226 11:55:09
{"level":"info","ts":1708970168.882816,"caller":"[email protected]/client.go:3048","msg":"run client"}
{"level":"info","ts":1708970168.9018764,"caller":"[email protected]/wrapper.go:658","msg":"<Error>","reqID":1,"errCode":162,"errString":"Historical Market Data Service error message:API historical data query cancelled: 1"}
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708938000, Open: 507.320000, High: 507.320000, Low: 507.220000, Close: 507.220000, Volume: 9.000000, Average: 507.275000, BarCount: 3>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708938060, Open: 507.260000, High: 507.260000, Low: 507.260000, Close: 507.260000, Volume: 3.000000, Average: 507.260000, BarCount: 2>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708938120, Open: 507.320000, High: 507.320000, Low: 507.310000, Close: 507.310000, Volume: 2.000000, Average: 507.314000, BarCount: 2>
....
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708969080, Open: 507.580000, High: 507.620000, Low: 507.470000, Close: 507.490000, Volume: 289.000000, Average: 507.527000, BarCount: 193>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708969140, Open: 507.480000, High: 507.520000, Low: 507.410000, Close: 507.470000, Volume: 460.000000, Average: 507.454000, BarCount: 276>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708969200, Open: 507.480000, High: 507.520000, Low: 507.410000, Close: 507.450000, Volume: 384.000000, Average: 507.464000, BarCount: 256>
HistoricalDataEnd. ReqId: 1, from 20240225 11:56:09 to 20240226 11:56:09
{"level":"info","ts":1708970228.895973,"caller":"[email protected]/client.go:3048","msg":"run client"}
{"level":"error","ts":1708970228.895973,"caller":"[email protected]/client.go:2942","msg":"write req error","nbytes":11,"reqMsg":"AAAABzI1ADEAMQA=","error":"short write","stacktrace":"github.com/hadrianl/ibapi.(*IbClient).goRequest\n\tC:/Users/chalu/go/pkg/mod/github.com/hadrianl/[email protected]/client.go:2942"}
{"level":"error","ts":1708970228.8965116,"caller":"[email protected]/client.go:3028","msg":"got client error in decode loop","error":"short write","stacktrace":"github.com/hadrianl/ibapi.(*IbClient).goDecode\n\tC:/Users/chalu/go/pkg/mod/github.com/hadrianl/[email protected]/client.go:3028"}
{"level":"info","ts":1708970228.9120264,"caller":"[email protected]/wrapper.go:658","msg":"<Error>","reqID":1,"errCode":162,"errString":"Historical Market Data Service error message:API historical data query cancelled: 1"}
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708938000, Open: 507.320000, High: 507.320000, Low: 507.220000, Close: 507.220000, Volume: 9.000000, Average: 507.275000, BarCount: 3>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708938060, Open: 507.260000, High: 507.260000, Low: 507.260000, Close: 507.260000, Volume: 3.000000, Average: 507.260000, BarCount: 2>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708938120, Open: 507.320000, High: 507.320000, Low: 507.310000, Close: 507.310000, Volume: 2.000000, Average: 507.314000, BarCount: 2>
....
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708969140, Open: 507.480000, High: 507.520000, Low: 507.410000, Close: 507.470000, Volume: 460.000000, Average: 507.454000, BarCount: 276>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708969200, Open: 507.480000, High: 507.520000, Low: 507.410000, Close: 507.450000, Volume: 384.000000, Average: 507.464000, BarCount: 256>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708969260, Open: 507.460000, High: 507.480000, Low: 507.420000, Close: 507.450000, Volume: 335.000000, Average: 507.455000, BarCount: 219>
HistoricalDataEnd. ReqId: 1, from 20240225 11:57:09 to 20240226 11:57:09
{"level":"info","ts":1708970288.8977444,"caller":"[email protected]/client.go:3048","msg":"run client"}
{"level":"error","ts":1708970288.8979797,"caller":"[email protected]/client.go:2942","msg":"write req error","nbytes":11,"reqMsg":"AAAABzI1ADEAMQA=","error":"short write","stacktrace":"github.com/hadrianl/ibapi.(*IbClient).goRequest\n\tC:/Users/chalu/go/pkg/mod/github.com/hadrianl/[email protected]/client.go:2942"}
{"level":"error","ts":1708970288.8985107,"caller":"[email protected]/client.go:3028","msg":"got client error in decode loop","error":"short write","stacktrace":"github.com/hadrianl/ibapi.(*IbClient).goDecode\n\tC:/Users/chalu/go/pkg/mod/github.com/hadrianl/[email protected]/client.go:3028"}
{"level":"info","ts":1708970348.8977444,"caller":"[email protected]/client.go:3048","msg":"run client"}
{"level":"error","ts":1708970348.8979797,"caller":"[email protected]/client.go:2942","msg":"write req error","nbytes":11,"reqMsg":"AAAABzI1ADEAMQA=","error":"short write","stacktrace":"github.com/hadrianl/ibapi.(*IbClient).goRequest\n\tC:/Users/chalu/go/pkg/mod/github.com/hadrianl/[email protected]/client.go:2942"}
{"level":"error","ts":1708970348.8985107,"caller":"[email protected]/client.go:3028","msg":"got client error in decode loop","error":"short write","stacktrace":"github.com/hadrianl/ibapi.(*IbClient).goDecode\n\tC:/Users/chalu/go/pkg/mod/github.com/hadrianl/[email protected]/client.go:3028"}
{"level":"info","ts":170897408.8977444,"caller":"[email protected]/client.go:3048","msg":"run client"}
{"level":"error","ts":170897408.8979797,"caller":"[email protected]/client.go:2942","msg":"write req error","nbytes":11,"reqMsg":"AAAABzI1ADEAMQA=","error":"short write","stacktrace":"github.com/hadrianl/ibapi.(*IbClient).goRequest\n\tC:/Users/chalu/go/pkg/mod/github.com/hadrianl/[email protected]/client.go:2942"}
{"level":"error","ts":170897408.8985107,"caller":"[email protected]/client.go:3028","msg":"got client error in decode loop","error":"short write","stacktrace":"github.com/hadrianl/ibapi.(*IbClient).goDecode\n\tC:/Users/chalu/go/pkg/mod/github.com/hadrianl/[email protected]/client.go:3028"}
As you can see, I get about 3 responses in 3 minutes, then nothing. I get that error traceback. I thought initially it was because of a rate limit issue, however I believe the API allows for up to 60 requests in 10 minutes, so 6 per minute, which I'm clearly under.