Accessing Market Data

Every instrument in the CryptoStruct Trading System has a unique Instrument ID that is consistent across exchanges. The Backend UI is the easiest way to look one up: browse the available instruments and pick out the IDs you need.

When you use the Strategy SDK, the Instrument ID is all you need. The Strategy Server connects to the right Market Data Adapter automatically and feeds the data into your strategy.

The rest of this page covers what's happening underneath. Worth understanding when:

• You are building a custom application that needs real-time market data.

• You are processing recorded market data with your own tools.

Accessing master data

The Backend UI is fine for poking at individual instruments by hand. For programmatic access at scale (or to feed instrument metadata into your own tools), use the Master Data API.

The Master Data API is a read-only HTTP interface for searching and filtering exchanges and instruments. The sections below are a practical intro, not a full reference.

Query all exchanges

http://<api-host>/api/exchanges

Query all instruments

http://<api-host>/api/instruments

Filtering instruments

Querying every instrument returns a large set. Filters narrow it down.

By exchange:

http://<api-host>/api/instruments?exchange_id=1

Open instruments only:

http://<api-host>/api/instruments?state=open

Active after a specific date:

http://<api-host>/api/instruments?active_after=2023-06-30

Finding market data hosts

Before subscribing to market data, you need to know which Market Data Adapter serves the instruments you care about.

Query the backend over HTTP:

http://<api-host>/api/v2/marketdata?instruments=<instrument-id-1>,<instrument-id-2>

Response structure

{
  "status": 200,
  "version": "4.9.0",
  "result": "success",
  "timestamp": "2023-06-08T12:11:12.507601Z",
  "cached": true,
  "count": 2,
  "data": [
    {
      "instrument_id": 22,
      "type": "perpetual",
      "code": "XBTUSD",
      "ticksize": 0.5,
      "decimals": 1,
      "lot_size": 100,
      "lot_decimals": 0,
      "exchange_id": 1,
      "exchange_code": "bitmex",
      "exchange_name": "BitMEX",
      "hosts": [
        {
          "host": "54.216.182.250",
          "ip": "54.216.182.250",
          "ipv6": null,
          "location": "eu-west-1",
          "port": 11003
        },
        {
          "host": "54.195.111.215",
          "ip": "54.195.111.215",
          "ipv6": null,
          "location": "eu-west-1",
          "port": 11003
        }
      ]
    },
    {
      "instrument_id": 67824,
      ...
    }
  ],
  "source": "http://masterdata.cryptostruct.com",
  "uri": "/api/v2/marketdata?instruments=22,67824",
  "cache_key": "PROXY::/api/v2/marketdata?instruments=22%2C67824"
}

The response carries:

• Basic master data (lot size, tick size).
• Multiple host addresses for fail-over and redundancy.

Subscribing to realtime data

Connecting via WebSocket

Pick any host from the /api/v2/marketdata response and connect over WebSocket:

ws://54.216.182.250:11003/api/v6

Once connected, log in and subscribe with the Market Data Protocol.

Multiple protocols and encodings

The example above hits the /api/v6 endpoint, which serves version 6 of the market-data protocol over JSON. That is the simplest setup. Other options exist:

• Protocol versions. Multiple versions can be live at once. New endpoints appear when the protocol is updated; previous versions stay reachable through their transition period.

• Encodings. Depending on the protocol version, SBE is also available alongside JSON, on dedicated endpoints.

• Domain sockets. When the Market Data Adapter runs on the same host as the client, Unix domain sockets bypass the kernel's TCP/IP stack and shave off a chunk of latency.

Querying available options

Ask the host what it supports:

http://54.216.182.250:11003/api/info?all=true

Response example:

{
    "version": "2.13.3-SNAPSHOT",
    "process_id": "bitmex-master",
    "capabilities": {
        "depthTopic": {
            "eventIdType": "ORDERED",
            "exchangeTimestampType": "UNKNOWN",
            "exchangeTimestampPrecision": "MILLIS"
        },
        "topOfBookTopic": {
            "eventIdType": "ORDERED",
            "exchangeTimestampType": "UNKNOWN",
            "exchangeTimestampPrecision": "MILLIS"
        },
        "tradesTopic": {
            "eventIdType": "UNORDERED",
            "exchangeTimestampType": "MATCHING_ENGINE",
            "exchangeTimestampPrecision": "MICROS"
        },
        "fundingRateTopic": null,
        "markPriceTopic": null,
        "indexPriceTopic": null,
        "crossTopicBookEventId": true,
        "predictedFundingRate": false
    },
    "endpoints": [
        {
            "endpoint": "/",
            "protocol_version": "1",
            "encoding": "JSON"
        },
        {
            "endpoint": "/api/v6",
            "protocol_version": "6",
            "encoding": "JSON",
            "domain_socket_path": "/tmp/bitmex-master_v6_json.socket"
        },
        {
            "endpoint": "/api/v6/sbe",
            "protocol_version": "6",
            "encoding": "SBE",
            "domain_socket_path": "/tmp/bitmex-master_v6_sbe.socket"
        }
    ]
}

The response carries:

• Available endpoints by protocol and encoding.
• Domain socket paths (for clients on the same host).
• Market-data capabilities (see Protocol for details).

Unix domain sockets

When the client process and the Market Data Adapter run on the same machine, the /api/info response includes a domain_socket_path. Use that path to open a domain-socket connection. The benefit is lower latency: domain sockets bypass the OS networking stack.

Protocol and framing

The protocol over a domain socket is the same as over WebSocket. Domain sockets, though, are raw byte streams (like TCP) with no built-in message boundary. To pull individual messages out of the stream, you need framing. See Binary Framing for details.

See Also

• Protocol. Market data protocol specification.
• SBE Encoding. Simple Binary Encoding for market data.