lightvector
Is there a roadmap? What is the plan ahead in 2025? thanks
I'm hoping to get a release out before too long with some minor bugfixes and such, and want to do some minor experiments with search features that could add small search quality improvements. There might be an larger network released this year, we'll see if it's enough better to compensate the slower speed. Other than that, I'm going to continue hosting the training at https://katagotraining.org/ and likely we'll do a learning rate cut this year to let the b28 optimize a bit better.
What do you think, do you have any further ideas or thoughts?
I'm hoping to get a release out before too long with some minor bugfixes and such, and want to do some minor experiments with search features that could add small search quality improvements. There might be an larger network released this year, we'll see if it's enough better to compensate the slower speed. Other than that, I'm going to continue hosting the training at https://katagotraining.org/ and likely we'll do a learning rate cut this year to let the b28 optimize a bit better.
What do you think, do you have any further ideas or thoughts?
@lightvector Thank you for your reply. In conjunction with your future project plan， I humbly suggest the next proposal：
Larger network is very valueable ， it might touch the edge of AI ability. Meanwhile it would cost more resource, people or company with ample GPU resources would harness it。 But it is not for everyone。 Will there be a plan for portable katago network and whole engine, so every mobile phone would be able to carry a katago engine with prediction quality not so bad. In summary, larger model achieves SOTA performance， it is terrific; and smaller and portable model with considerable performance is also in urgent need of 。
@jackiesteed Do u just want a mobile version? Well, KataGo is written in C++, so it's cross-platform, you can compile. But none official arm64 release yet. However, many GUIs has done. For example, if you have a android phone, you can search aki65 BadukAI. And perhaps the README of KataGo could add some arm64 contents.
@jackiesteed Do u just want a mobile version? Well, KataGo is written in C++, so it's cross-platform, you can compile. But none official arm64 release yet. However, many GUIs has done. For example, if you have a android phone, you can search aki65 BadukAI. And perhaps the README of KataGo could add some arm64 contents.
Not just a mobile version。 maybe it should be called the best performance on limit gpu resource。
@jackiesteed But is far more than sufficient already. With
- KG b15/20 neural network models
- recent mobile chips, A14 or 8gen3
- around 2 seconds per move
You can beat any professional Go/Baduk/Weiqi player to the ground. chips AI TOPS
chips AI TOPS
@HackYardo Very encouraged to hear that。 So it comes to the application stage。 Maybe one of future target of katago is to be compatable with most mainstream devices。
Even on weak hardware, it may often be the case that the largest neural network is the best, due to being so much stronger than the faster and smaller networks that it more than makes up for the loss in performance. I ran a test:
Networks used:
kata1-b10c128-s1141046784-d204142634.txt.gz
kata1-b15c192-s1672170752-d466197061.bin.gz
kata1-b20c256x2-s5303129600-d1228401921.bin.gz
kata1-b40c256-s12350780416-d3055274313.bin.gz
kata1-b18c384nbt-s9996604416-d4316597426.bin.gz
kata1-b28c512nbt-s8326494464-d4628051565.bin.gz
Big match testing all networks at 1, 4, 8, 16, 32, 64, 128, 256 visits per move, default search and temperature settings.
(edit: But also only 1 search thread for all searches)
Starting positions were randomly chosen from all game states occurring within the first 15 moves in the
Go4Go pro game collection from all of history up through sometime mid 2022, so often seeding 1-2 corners
with a wide variety of human-played joseki, but also leaving plenty of bare 3-4 and 4-4 points for the bots to also
play out what sequences they like.
Rules randomized between all the various rules, komi set to approximately fair *given* the rules and initial board
position of each particular game.
Elos (+/- one approx standard error):
b28nbt-v256 : 1615.88 +/- 78.01
b28nbt-v128 : 1326.52 +/- 47.62
b18nbt-v256 : 1274.28 +/- 45.26
b40res-v256 : 1110.24 +/- 38.57
b28nbt-v64 : 1105.16 +/- 38.15
b18nbt-v128 : 1060.67 +/- 37.26
b18nbt-v64 : 864.91 +/- 33.04
b28nbt-v32 : 809.27 +/- 32.55
b40res-v128 : 807.40 +/- 32.65
b20res-v256 : 665.78 +/- 30.86
b40res-v64 : 653.28 +/- 30.35
b18nbt-v32 : 572.88 +/- 29.61
b28nbt-v16 : 486.09 +/- 28.21
b20res-v128 : 430.79 +/- 28.05
b40res-v32 : 401.58 +/- 27.36
b18nbt-v16 : 336.78 +/- 26.42
b20res-v64 : 276.92 +/- 25.41
b28nbt-v8 : 209.79 +/- 24.41
b15res-v256 : 165.66 +/- 24.80
b40res-v16 : 145.66 +/- 23.63
b20res-v32 : 59.32 +/- 23.05
b15res-v128 : 22.51 +/- 23.16
b18nbt-v8 : 13.97 +/- 22.88
b28nbt-v4 : -17.77 +/- 22.55
b15res-v64 : -79.28 +/- 22.25
b40res-v8 : -91.08 +/- 21.99
b28nbt-v1 : -111.05 +/- 22.13
b20res-v16 : -134.01 +/- 22.09
b18nbt-v4 : -184.28 +/- 21.87
b10res-v256 : -207.33 +/- 22.42
b40res-v4 : -266.55 +/- 22.51
b15res-v32 : -275.48 +/- 22.67
b18nbt-v1 : -285.71 +/- 22.64
b10res-v128 : -319.31 +/- 22.78
b20res-v8 : -354.49 +/- 22.87
b40res-v1 : -375.28 +/- 23.35
b10res-v64 : -422.72 +/- 23.81
b15res-v16 : -479.95 +/- 24.77
b20res-v4 : -545.30 +/- 25.95
b20res-v1 : -587.52 +/- 26.48
b10res-v32 : -619.22 +/- 26.92
b15res-v8 : -719.11 +/- 29.52
b10res-v16 : -796.33 +/- 31.51
b15res-v4 : -884.08 +/- 34.02
b15res-v1 : -996.38 +/- 37.28
b10res-v8 : -1042.91 +/- 39.24
b10res-v4 : -1287.50 +/- 50.83
b10res-v1 : -1350.38 +/- 55.37
Some points of comparison regarding the test results: The current b28...
- with 8 visits appears to be close or slightly better than the old 15-block net with 256 visits.
- with 32 visits is clearly better than the old 20 block net with 256 visits.
- with 64 visits is similar to the old 40 block net with 256 visits.
So if you just care about strength alone, maybe this can give some guidelines as to what level of performance factor you need for the larger network to be better than the smaller one anyways. Although it's also reasonable to having a faster network anyways even in some cases where it's a little weaker (e.g. more visits even if weaker can make it easier to see evaluations for a larger number of possible moves).
And via b✕c✕c speed formula from LeelaChessZero userguide, I add z axis to see how many evaluations at equal time.
One idea about search.
The difference of variations between KataGo's and professional explanation's is "always global" or "assumption-base". Professionals assume that this stone should run away or you will lose, then show conclusions using variations. Humans who know assumption can understand it.
Is it possible to impose such manually given assumptions/constraints on KataGo's search? If so, I think that KataGo which cares assumptions will be able to show nicer variations especially for local life-death or tesuji parts on global position.
Though I am not sure if this idea is well-defined or not, I wrote this comment as brainstorming.
If the network "knows" some information in the search tree, it can probably improve overall performance? Not sure what information should be given to the network.
@ChinChangYang san,
Assumptions or constraints, such as "this stone should run away", will be given by humans. They are not correct information. Humans often assume such wrong constraints and want to know conclusions.
KataGo network tends to avoid such wrong constraints themselves and cannot make variations for them, I think.
@y-ich I think you are trying to talk about the horizon effect: a limitation of lookahead in minimax-based search algorithms, where an important event is artificially postponed beyond the search depth, making it invisible to the evaluation function.
To mitigate the horizon effect, we may provide the network with additional information about the root node of the search tree, aiming to probably help it recognize which stones are effectively dead or alive.
This is just my idea. I haven't experimented with this.
@ChinChangYang san,
KataGo may sometimes avoid some dead or alive problem by horizon effect, but usually avoids the ones because it is small yet compared with global positions. But humans want to know the result of this small part.
@y-ich KaTrain supports a region of interest for this purpose.
https://youtu.be/tXniX57KtKk?si=16zdQXTb8zx_zDOn
@ChinChangYang san,
Suppose that some one want to know the result of a ladder. KataGo does not show variation for it. And I guess that a region of interest also does not solve it. So I think that assumption/constraint is not same as a region of interest.
You can enjoy your idea, not mine:)
@ChinChangYang san,
Suppose that some one want to know the result of a ladder. KataGo does not show variation for it. And I guess that a region of interest also does not solve it. So I think that assumption/constraint is not same as a region of interest.
You can enjoy your idea, not mine:)
@y-ich “A Master of Go” the app showed some experience of that.
A very quick experiment for the idea "this stone should run away". It's interesting that I needed to combine humanSL 9k policy to solve a capturing problem.
https://github.com/kaorahi/visual_MCTS/tree/master/sample4
@jackiesteed san,
Thank you for using the app! I am a developer of "A Master of Go". As you pointed out, it shows ladder reading when you use the default engine, which is based on Leela Zero. Leela Zero or Alpha Zero reads ladders in MCTS variations, so I can visualized them.
KataGo knows the results of ladders in inputs to NN, and it does not read ladders as variations. However, if you give KataGo some constraint, KataGo will show the result, I think.
@jackiesteed san,
Thank you for using the app! I am a developer of "A Master of Go". As you pointed out, it shows ladder reading when you use the default engine, which is based on Leela Zero. Leela Zero or Alpha Zero reads ladders in MCTS variations, so I can visualized them.
KataGo knows the results of ladders in inputs to NN, and it does not read ladders as variations. However, if you give KataGo some constraint, KataGo will show the result, I think.
@y-ich san,
Awesome app “A Master of Go” and thank you。 Amazing 170 playouts per second! I saw this conclusion on the homepage：https://new3rs.github.io/a_master_of_go/index.html
Can you share how you do the benchmark? thank you.😄
@jackiesteed san,
Thank you for your comment but it seems off topic in this thread. If you are really interested in it, which is a very old promotion page, please e-mail to me via the app.