bitshares-explorer-api icon indicating copy to clipboard operation
bitshares-explorer-api copied to clipboard
verified publisher icon bitshares

Optimizations for api_explorer_get_top_proxies (#51)

Open Zapata opened this issue 6 years ago • 2 comments

  • add start and limit parameters to /top_proxy
  • add /proxy_count
  • parallelize ES accounts retrieval

Zapata avatar Aug 03 '19 19:08 Zapata

To fix #51

Zapata avatar Aug 03 '19 19:08 Zapata

When i try to use any of the involved calls in localhost i am getting the following error in the console after about a minute:

(wrappers_env) alfredo@alfredo-System-Product-Name:~/PycharmProjects/pull60/bitshares-explorer-api$ flask run --host=0.0.0.0
 * Tip: There are .env or .flaskenv files present. Do "pip install python-dotenv" to use them.
 * Serving Flask app "app.py"
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
INFO:werkzeug: * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
WARNING:elasticsearch:GET https://elasticsearch.bitshares-kibana.info:443/objects-account/_search?scroll=5m&size=1000 [status:N/A request:0.088s]
Traceback (most recent call last):
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 217, in perform_request
    method, url, body, retries=Retry(False), headers=request_headers, **kw
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 641, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/urllib3/util/retry.py", line 344, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 379, in _make_request
    httplib_response = conn.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1121, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 438, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 394, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 772, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 659, in read
    v = self._sslobj.read(len)
SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1941)
WARNING:elasticsearch:GET https://elasticsearch.bitshares-kibana.info:443/objects-account/_search?scroll=5m&size=1000 [status:N/A request:0.277s]
Traceback (most recent call last):
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 217, in perform_request
    method, url, body, retries=Retry(False), headers=request_headers, **kw
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 641, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/urllib3/util/retry.py", line 344, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 379, in _make_request
    httplib_response = conn.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1121, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 438, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 394, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 772, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 659, in read
    v = self._sslobj.read(len)
SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:1941)
29 accounts to load via websocket
ERROR:bitshares-explorer-api:Exception on /top_proxies [GET]
Traceback (most recent call last):
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/connexion/decorators/decorator.py", line 48, in wrapper
    response = function(request)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/connexion/decorators/uri_parsing.py", line 143, in wrapper
    response = function(request)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/connexion/decorators/validation.py", line 347, in wrapper
    return function(request)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/connexion/decorators/parameter.py", line 126, in wrapper
    return function(**kwargs)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/api/explorer.py", line 414, in get_top_proxies
    proxies = _get_all_proxies()
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/flask_caching/__init__.py", line 795, in decorated_function
    rv = f(*args, **kwargs)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/api/explorer.py", line 392, in _get_all_proxies
    holders = _get_holders()
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/flask_caching/__init__.py", line 795, in decorated_function
    rv = f(*args, **kwargs)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/api/explorer.py", line 457, in _get_holders
    accounts = _load_missing_accounts_via_ws(account_ids, accounts)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/api/explorer.py", line 443, in _load_missing_accounts_via_ws
    missing_accounts = _get_accounts_by_chunks_via_ws(accounts_ids_to_load)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/api/explorer.py", line 434, in _get_accounts_by_chunks_via_ws
    accounts = bitshares_ws_client.request('database', 'get_accounts', [ account_ids[i:i+chunk_size] ])
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/services/bitshares_websocket_client.py", line 24, in request
    return self._safe_request(api, method_name, params)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/services/bitshares_websocket_client.py", line 43, in _safe_request
    self.ws.send(request_string)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/websocket/_core.py", line 253, in send
    return self.send_frame(frame)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/websocket/_core.py", line 278, in send_frame
    l = self._send(data)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/websocket/_core.py", line 448, in _send
    return send(self.sock, data)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/websocket/_socket.py", line 157, in send
    return _send()
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/websocket/_socket.py", line 139, in _send
    return sock.send(data)
  File "/usr/lib/python2.7/ssl.py", line 725, in send
    v = self._sslobj.write(data)
error: [Errno 32] Broken pipe
INFO:werkzeug:127.0.0.1 - - [23/Aug/2019 19:09:02] "GET /top_proxies?limit=10 HTTP/1.1" 500 -

Other ES calls work fine.

oxarbitrage avatar Aug 23 '19 22:08 oxarbitrage