[SVS]: fix: fix visinger2 inference

[South-Twilight]

What did you change?

fix visinger2 inference: support phoneme duration inference.

---

Why did you make this change?

fix bug.

---

Is your PR small enough?

yes

---

Additional Context

Related to #6292

[codecov[bot]]

Codecov Report

:x: Patch coverage is 87.50000% with 1 line in your changes missing coverage. Please review. :white_check_mark: Project coverage is 69.39%. Comparing base (0459b11) to head (3f3d5e3). :warning: Report is 463 commits behind head on master.

Files with missing lines	Patch %	Lines
espnet2/gan_svs/vits/generator.py	87.50%	1 Missing :warning:

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #6306       +/-   ##
===========================================
+ Coverage   56.24%   69.39%   +13.14%     
===========================================
  Files         897      759      -138     
  Lines       84917    69851    -15066     
===========================================
+ Hits        47765    48474      +709     
+ Misses      37152    21377    -15775     

Flag	Coverage Δ
test_integration_espnet2	46.96% <ø> (+1.06%)	:arrow_up:
test_integration_espnetez	?
test_python_espnet2	62.07% <87.50%> (+11.17%)	:arrow_up:
test_python_espnet3	16.06% <0.00%> (?)
test_python_espnetez	?
test_utils	62.07% <87.50%> (+43.30%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sw005320]

thanks!