qlib icon indicating copy to clipboard operation
qlib copied to clipboard
verified publisher icon microsoft

Get poor result when i run KRNN .etc deep learning models

Open LoserFeng opened this issue 4 months ago • 1 comments

I get poor score and the score did not improve after traning , did anyone get the same result with me ,i try the newest qlib and not motify any code yet

I run:

qrun examples/benchmarks/KRNN/workflow_config_krnn_Alpha360.yaml


[76332:MainThread](2025-10-07 14:04:22,107) WARNING - qlib.utils - [__init__.py:849] - The parameter `reweighter` with value `None` is ignored.
[76332:MainThread](2025-10-07 14:04:23,075) INFO - qlib.KRNN - [pytorch_krnn.py:458] - training...
[76332:MainThread](2025-10-07 14:04:23,076) INFO - qlib.KRNN - [pytorch_krnn.py:462] - Epoch0:
[76332:MainThread](2025-10-07 14:04:23,076) INFO - qlib.KRNN - [pytorch_krnn.py:463] - training...
[76332:MainThread](2025-10-07 14:04:26,092) INFO - qlib.KRNN - [pytorch_krnn.py:465] - evaluating...
[76332:MainThread](2025-10-07 14:04:27,237) INFO - qlib.KRNN - [pytorch_krnn.py:468] - train -0.996956, valid -0.996785
[76332:MainThread](2025-10-07 14:04:27,251) INFO - qlib.KRNN - [pytorch_krnn.py:462] - Epoch1:
[76332:MainThread](2025-10-07 14:04:27,251) INFO - qlib.KRNN - [pytorch_krnn.py:463] - training...
[76332:MainThread](2025-10-07 14:04:29,450) INFO - qlib.KRNN - [pytorch_krnn.py:465] - evaluating...
[76332:MainThread](2025-10-07 14:04:30,540) INFO - qlib.KRNN - [pytorch_krnn.py:468] - train -0.996870, valid -0.996965
[76332:MainThread](2025-10-07 14:04:30,540) INFO - qlib.KRNN - [pytorch_krnn.py:462] - Epoch2:
[76332:MainThread](2025-10-07 14:04:30,542) INFO - qlib.KRNN - [pytorch_krnn.py:463] - training...
[76332:MainThread](2025-10-07 14:04:32,669) INFO - qlib.KRNN - [pytorch_krnn.py:465] - evaluating...
[76332:MainThread](2025-10-07 14:04:33,792) INFO - qlib.KRNN - [pytorch_krnn.py:468] - train -0.996681, valid -0.996343
[76332:MainThread](2025-10-07 14:04:33,793) INFO - qlib.KRNN - [pytorch_krnn.py:462] - Epoch3:
[76332:MainThread](2025-10-07 14:04:33,794) INFO - qlib.KRNN - [pytorch_krnn.py:463] - training...
[76332:MainThread](2025-10-07 14:04:35,987) INFO - qlib.KRNN - [pytorch_krnn.py:465] - evaluating...
[76332:MainThread](2025-10-07 14:04:37,151) INFO - qlib.KRNN - [pytorch_krnn.py:468] - train -0.996362, valid -0.996110
[76332:MainThread](2025-10-07 14:04:37,153) INFO - qlib.KRNN - [pytorch_krnn.py:462] - Epoch4:
[76332:MainThread](2025-10-07 14:04:37,153) INFO - qlib.KRNN - [pytorch_krnn.py:463] - training...
[76332:MainThread](2025-10-07 14:04:39,335) INFO - qlib.KRNN - [pytorch_krnn.py:465] - evaluating...
[76332:MainThread](2025-10-07 14:04:40,448) INFO - qlib.KRNN - [pytorch_krnn.py:468] - train -0.996215, valid -0.995404
[76332:MainThread](2025-10-07 14:04:40,449) INFO - qlib.KRNN - [pytorch_krnn.py:462] - Epoch5:
[76332:MainThread](2025-10-07 14:04:40,449) INFO - qlib.KRNN - [pytorch_krnn.py:463] - training...
[76332:MainThread](2025-10-07 14:04:42,630) INFO - qlib.KRNN - [pytorch_krnn.py:465] - evaluating...
[76332:MainThread](2025-10-07 14:04:43,747) INFO - qlib.KRNN - [pytorch_krnn.py:468] - train -0.996109, valid -0.995462

LoserFeng avatar Oct 07 '25 06:10 LoserFeng

Hi @Yzmblog! The negative loss values you're seeing are actually expected behavior for the KRNN model in Qlib. Let me explain why:

Understanding the Loss Values

The KRNN (K-Nearest Neighbor Neural Network) model in Qlib uses negative correlation as the loss metric, which is why you're seeing values like -0.996965.

Why Negative Loss?

  1. Correlation-based Loss Function: KRNN optimizes for maximum correlation between predictions and actual returns
  2. Negative Sign Convention: The loss is defined as -correlation so that minimizing loss = maximizing correlation
  3. Your Results Are Actually Good:
    • Loss of -0.996965 means correlation of +0.996965 (99.7% correlation!)
    • This indicates your model is learning very well

Interpreting Your Training Log

train -0.996965, valid -0.996965  # Epoch 1: 99.7% correlation
train -0.996681, valid -0.996343  # Epoch 2: Still ~99.6% correlation
train -0.996362, valid -0.996110  # Epoch 3: Maintaining high correlation

What This Means:

✅ Good Signs:

  • Very high correlation (>99%) between predictions and targets
  • Stable training and validation metrics
  • Model is converging properly

⚠️ Potential Concerns:

  • Suspiciously high correlation - This might indicate:
    1. Look-ahead bias - Using future information in features
    2. Data leakage - Target variable leaking into features
    3. Overfitting - Model memorizing training patterns

Recommended Actions

1. Check for Data Leakage

Verify your feature engineering doesn't include future information:

# Bad - uses future data
df['future_return'] = df['close'].shift(-1) / df['close'] - 1

# Good - uses only past data
df['past_return'] = df['close'] / df['close'].shift(1) - 1

2. Validate on Out-of-Sample Data

Test on completely unseen data:

from qlib.contrib.evaluate import backtest

# Use a future time period not in training
pred_df = model.predict(dataset_test)
report, positions = backtest(pred_df, strategy_config)

print(report)  # Check actual trading performance

3. Check Feature-Target Correlation

Ensure no single feature has >95% correlation with target:

import pandas as pd

# Load your data
data = dataset.prepare("train")
features = data[0]  # X
targets = data[1]   # y

# Check correlations
correlations = pd.DataFrame(features).corrwith(pd.Series(targets))
print(correlations.sort_values(ascending=False))

# Flag suspicious features
suspicious = correlations[correlations.abs() > 0.95]
if len(suspicious) > 0:
    print(f"⚠️ Warning: {len(suspicious)} features have >95% correlation with target")
    print(suspicious)

4. Use Proper Cross-Validation

Ensure time-series aware splitting:

from qlib.contrib.data.dataset import TSDatasetH

dataset = TSDatasetH(
    handler=handler,
    segments={
        "train": ("2015-01-01", "2018-12-31"),
        "valid": ("2019-01-01", "2019-12-31"),
        "test": ("2020-01-01", "2020-12-31")  # Completely unseen
    }
)

Expected Realistic Performance

For stock prediction tasks, typical correlation values should be:

  • Training: 0.05 - 0.20 (5-20%)
  • Validation: 0.03 - 0.15 (3-15%)
  • Test: 0.02 - 0.10 (2-10%)

Your 99%+ correlation is extremely unusual and almost certainly indicates a data issue rather than exceptional model performance.

Next Steps

  1. Review your data preprocessing pipeline
  2. Check the Qlib example notebooks for proper feature engineering
  3. Verify no target leakage in your Alpha158 or custom features
  4. Test on a completely held-out test set

Would you be able to share your data handler configuration? That would help identify the specific issue.

Hope this helps clarify the situation!

Dex947 avatar Oct 09 '25 10:10 Dex947