repeng icon indicating copy to clipboard operation
repeng copied to clipboard

Add qualitative performance validation

Open wassname opened this issue 5 months ago • 2 comments

This helps find out which methods work best, which models it works with, and so on. For example this show that middle layers work best, it only works on model >4B, and diff_pca is best

wassname avatar Sep 21 '25 06:09 wassname

it only works on model >4B

Just in case: have you checked wether the error message Error at layer 27: 'Qwen3DecoderLayer' object has no attribute 'set_control' repeated over and over at the bottom is an issue here?

thiswillbeyourgithub avatar Sep 21 '25 10:09 thiswillbeyourgithub

oh yeah that is just because those layers were not transformed into repeng control blocks. And I just used a try except pattern over all layers

With small models, they are kind of incoherent to start with, and I suspect they don't have well-developed inner concepts, making them harder to steer.

wassname avatar Sep 21 '25 20:09 wassname