oklog
What's the best way to test the cluster in large installation is working
Hi, I have the following large installation (2 ingests and 3 stores) started as follows:
Commands: oklog-0.3.2-linux-amd64 ingest -cluster 172.27.47.21 -peer 172.27.47.21 -peer 172.27.47.22 -peer 172.27.47.23 -peer 172.27.47.24 -peer 172.27.47.25
oklog-0.3.2-linux-amd64 ingest -cluster 172.27.47.22 -peer 172.27.47.21 -peer 172.27.47.22 -peer 172.27.47.23 -peer 172.27.47.24 -peer 172.27.47.25
oklog-0.3.2-linux-amd64 store -cluster 172.27.47.23 -peer 172.27.47.21 -peer 172.27.47.22 -peer 172.27.47.23 -peer 172.27.47.24 -peer 172.27.47.25
oklog-0.3.2-linux-amd64 store -cluster 172.27.47.24 -peer 172.27.47.21 -peer 172.27.47.22 -peer 172.27.47.23 -peer 172.27.47.24 -peer 172.27.47.25
oklog-0.3.2-linux-amd64 store -cluster 172.27.47.25 -peer 172.27.47.21 -peer 172.27.47.22 -peer 172.27.47.23 -peer 172.27.47.24 -peer 172.27.47.25
Output: First ingest keeps showing the following output until the second ingest joins the cluster: level=warn component=cluster NumMembers=1 msg="I appear to be alone in the cluster"
Second ingest shows the following output: level=info ingest_path=data/ingest
First store keeps showing the following output until the second store joins the cluster: ts=2018-08-02T13:06:04.503169534Z level=warn component=Consumer op=gather warning="replication factor 2, available peers 1: replication currently impossible"
Second and third stores show the following output (respectively): ts=2018-08-02T13:07:44.610961417Z level=info StoreLog=data/store ts=2018-08-02T13:08:08.235069864Z level=info StoreLog=data/store
Testing cluster: When I use testsvc to test the service I get output like the following - although I haven’t hooked the cluster with any application or source of logs -
2018-08-02T14:26:32+01:00 foo 000000100 SBE2 0T30 0Y43 39E4 4Q7B TK7N VVTR K2HG VSKS Z8P6 R3A8 D49G FGWD 6QH2 A9Y7 41JJ 708W 6TGM 5RZQ AG4J ZGDJ JQVR PZVN ZZ8W A6WF ZTK0 0MBT WPH6 E5DH 3APC 58K8 KJMM 25GX Y440 HCWR SJ4D M8BG S21B 2B1M 1NAB XM1J 4D7Z 0QZZ 220Z QM5E 2BFN B216 4HM7 CMQN AXVT HEJ0 XGHV 17S8 WQXS M23N 55F6 RZHT XG9D 72AJ 9DGW S8NS 6RSV T2A1 FDJ2 771N 6HMQ WFQN 1KY3 3TD6 0DRW 2WWJ 1TGF CQ6W 8EMB B030 2TG2 K3Z2 Z9HQ DE1P HPPK BCZV SBBH 2RKD 6S16 DR8J P7DS 26YB Y4KC 4X8T 6E18 DGHE 8CDA 4KRG PA8W N 2018-08-02T14:26:32+01:00 foo 000000101 01SF32MTSYYSDGM 8N089YN3DFTGK0W RNV9BJ1Q03130GE FDEH838MRXE1PY4 DJKRTK0K1YD0BW3 TCVTZP4Q9SHEPRS ZDBE2N09XWZ3CDG JT1KJ9F8EPKJSYX W8K2EGX034KPZS9 8RS8QZ9GPVAMN24 MWM990KTNSSJEH5 T6VWGA30Z4SWCF6 DT8EKVC1E105BZX G1SG4TYCDKK4C1A GA545A0EK52MMSK THZXVHVV9DS8ER7 A1MD9Q4B93ED3X3 895GSW94RDQSCH0 Z05D63ZJKG8ZPFW RTKGT2VV5PC9NM1 85MQC6SG408DPKT V94F94H7B4YYTX0 GJ4E7YJPAJSG3TJ F6T6H3D79QBYVQZ MQY4EXTYDSW2AKC QY9NWDPYB4A30ZF DZF0NT2F7W056KN PDFPV5RDBCHW0V9 XFSBHZ65V497TQ2 ZD62Z4R 2018-08-02T14:26:33+01:00 foo 000000102 8XN18 E1HZC 6TS9H BPEP3 ARMCE CV25Y 0Q69D PFDHQ 2CDC6 Z708X Z3BFN EF20N Y2VP5 PAANT BW4WN EA8HJ 2SFX7 9BZ1V FD4VC 1A4ZF 1PP3W PBZJC 2B11Z GD0QK 419YZ Y0X2T BBJ2B ACX9Y G7X1W 4QN1Q CXNP9 JQAKB RETB7 0C6DM DBXSG 9MAVT SEFJN 1286S 5BY06 JP8SC XSAQD TYT3B T5FK7 JQSXX FNE53 DN71G 1R8J3 QSBTN HM7A2 PKAMV C6J5H CJKYT AHQK1 3SMJJ YTVT6 4AP5N 3RB3P 1WKS6 CP7FC BB7P4 VZ7QQ FT3JS 27T71 90F97 K3JSW XX6KA 8WJEC YQ5T4 719J9 Z5425 NK04S M966W 1VCM7 27TMX 8BHB6 090VN 0108D NNR48 JPH 2018-08-02T14:26:33+01:00 foo 000000103 J5S8CS5GJJNMQRK 41M7KPB5S8ZY686 SYT1ZE40XSETGN6 G608YKWF05C4EAK GBEVQEJWE6M3MYZ W1AGX1QE0NT47V4 91VJQPRRVG61AJM MTRTKJSPY98HMW2 QNHCEZ9FJBT93NV GEX84DTXNFHJW7T 7HRJ9MT6NK5AKQQ PN9E9QDN002M5TQ 4Z52WC0JMB491DV ZPE3RCSKK0XKTC8 0BPMVC63K9J8ZGS YEKGB1P84DYJM8W XHJ8TD31MRS339D QC2N7285DS17SP1 RHNH7NZHGGV7C1G VN9KBXR5S4MQF8A 6929BYWCFE5W8GH 7S26H0TAP8H15XJ F17MCQTQDJSG1ZF 71A6E7SVJZB2JZV XWWPYH809F2FWZG 4XVM7Y88Z4KQSH6 YRWG31Y41BRS50N 1N12451GAM8CXCH QA62Q85H7J0JHN0 2WT4BGG 2018-08-02T14:26:33+01:00 foo 000000104 S6YJ4SDHNDNMCCZ3 MX86GFKJCC921JPA B710CFZVY5HE2HHT RZP0AMF0A7AJDDFK S9YVY0SHS0HXY56H VAXHE1DHZFNBQ7NY 5ZAC1E70MKPQAM39 PKTVTENGJB5XY91P F674EXK1464CE8TV Q833271WAYP6PX1T 1RRKZXMZSCZ1MH5V R2TK9E35NP7B7VW3 4WQF328Y7GSSYZEH TQ48GYQG7SQK3774 MCCZRSANK695A020 7C08P0VY2CVJ2719 14Z9RAC5736CRZB9 KCRQ822P39ARB2YB HM6RSVAXP17SHYTV Q65QGX2W2SD37MT6 MYN96N87Q1XK0P5X 9SA0G4VGNS5MTA6R 57E6JVWC7EKERHV8 Z2K5RAEJP8F2YE7C WYKAMRW91YDW52QZ 93F320FVWJA5061X Z1NTG5W9YK33ZR5T H3GA9G5AYBWF
So my question is: what is the best way to test the cluster is working before forwarding any logs to it (using output or a test or member list command) since the testsvc output doesn't seem to be accurate in my opinion (unless there's something wrong I am doing).
Thanks, Hoch
