feat(recipe): Add support YAML recipe

[Geod24]

This makes dub recognize dub.yaml as a configuration file format. The code is very similar to the JSON one, as Dub has been using the YAML parser for multiple years. It allows us to use a configuration format that is in widespread use and is more human-oriented than JSON.

[github-actions[bot]]

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 0

Total warnings: 0

Build statistics:

 statistics (-before, +after)
-executable size=5105200 bin/dub
-rough build time=59s
+executable size=5154440 bin/dub
+rough build time=60s

Full build output

DUB version 1.39.0, built on Mar 20 2025
LDC - the LLVM D compiler (1.40.1):
  based on DMD v2.110.0 and LLVM 19.1.7
  built with LDC - the LLVM D compiler (1.40.1)
  Default target: x86_64-unknown-linux-gnu
  Host CPU: znver3
  http://dlang.org - http://wiki.dlang.org/LDC


  Registered Targets:
    aarch64     - AArch64 (little endian)
    aarch64_32  - AArch64 (little endian ILP32)
    aarch64_be  - AArch64 (big endian)
    amdgcn      - AMD GCN GPUs
    arm         - ARM
    arm64       - ARM64 (little endian)
    arm64_32    - ARM64 (little endian ILP32)
    armeb       - ARM (big endian)
    avr         - Atmel AVR Microcontroller
    bpf         - BPF (host endian)
    bpfeb       - BPF (big endian)
    bpfel       - BPF (little endian)
    hexagon     - Hexagon
    lanai       - Lanai
    loongarch32 - 32-bit LoongArch
    loongarch64 - 64-bit LoongArch
    mips        - MIPS (32-bit big endian)
    mips64      - MIPS (64-bit big endian)
    mips64el    - MIPS (64-bit little endian)
    mipsel      - MIPS (32-bit little endian)
    msp430      - MSP430 [experimental]
    nvptx       - NVIDIA PTX 32-bit
    nvptx64     - NVIDIA PTX 64-bit
    ppc32       - PowerPC 32
    ppc32le     - PowerPC 32 LE
    ppc64       - PowerPC 64
    ppc64le     - PowerPC 64 LE
    r600        - AMD GPUs HD2XXX-HD6XXX
    riscv32     - 32-bit RISC-V
    riscv64     - 64-bit RISC-V
    sparc       - Sparc
    sparcel     - Sparc LE
    sparcv9     - Sparc V9
    spirv       - SPIR-V Logical
    spirv32     - SPIR-V 32-bit
    spirv64     - SPIR-V 64-bit
    systemz     - SystemZ
    thumb       - Thumb
    thumbeb     - Thumb (big endian)
    ve          - VE
    wasm32      - WebAssembly 32-bit
    wasm64      - WebAssembly 64-bit
    x86         - 32-bit X86: Pentium-Pro and above
    x86-64      - 64-bit X86: EM64T and AMD64
    xcore       - XCore
    xtensa      - Xtensa 32
   Upgrading project in /home/runner/work/dub/dub/
    Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.40.1/x64/ldc2-1.40.1-linux-x86_64/bin/ldc2 for x86_64.
    Building dub 1.39.0-rc.1+commit.54.g32d6793a: building configuration [application]
     Linking dub
STAT:statistics (-before, +after)
STAT:executable size=5154440 bin/dub
STAT:rough build time=60s

[Geod24]

Note that I did mention it at a DLF meeting a while back.

[s-ludwig]

I remember calling you out on this possible plan when you first integrated Configy. Now, the main argument that has been made in the big community outcry back when the JSON/SDL topic came up - namely fragmenting the package format landscape and adding maintenance burden - is still standing. What has lead people to change their mind on this? Back then there were very strong feelings associated with this.

Now, admittedly, I don't quite agree with the gravity of the fragmentation argument. It's more so that this whole discussion, which has been brought up time and time again, has cost me a lot of motivation and, apart from the objective aspects, it still bothers me a bit to now see someones favorite color of the shed just being sneaked in.

But regarding YAML itself, I really think the language design is flawed in many ways and not a technically good choice. The main argument is that it's frequently used for CI configuration files and in some other more or less related areas. But in terms of usability and simplicity, SDL is just superior for this use case.

The fact that something is technically easy to implement alone is a very weak argument when it comes to designing a system - no matter whether Confiy just happens to be able to parse YAML or DMD conveniently has C bitfield semantics already implemented.

Anyway, apart from all of this, this is a change with a strong long-term impact that IMO warrants a proper community discussion. And if we go ahead with this, there is also quite a bit of documentation to be amended (and maintained, in case of new recipe features).

[Geod24]

I remember calling you out on this possible plan when you first integrated Configy. Now, the main argument that has been made in the big community outcry back when the JSON/SDL topic came up - namely fragmenting the package format landscape and adding maintenance burden - is still standing. What has lead people to change their mind on this? Back then there were very strong feelings associated with this.

Interestingly, things went the other way around for Configy. I ended up abstracting away the YAML dependency, and you can now use it to parse SDL, JSON, TOML, as long as you have a backend for it. I actually tried to remove the YAML dependency in Dub as a result (see the first commit of #3023 for the JSON backend), but I was not getting column information (only lines) so it was a bit of a regression.

For further reference, Configy was introduced in #2280 . My stance at the time was against adding YAML support: https://github.com/dlang/dub/pull/2280#issuecomment-1161558740 I think most of our discussion about YAML happened a bit later, in #2546 . I went through the thread on the forum again, where I request community feedback, and there was a lot of discussions, but little conclusion. The one thing I want to highlight is Walter's post:

I don't know much technically about the various formats, but on a non-technical basis:

1. We win if we go with a well-known, popular format. We are not in the business of promoting an obscure format or inventing our own. We picked github and git long ago simply because it was the most popular, and that decision paid off handsomely for us. As you wrote, it minimizes user friction.

2. We need a quality import/export mechanism for it written in D with an easy license (preferably Boost).

I'm on board with whatever you choose.

Additionally, @atilaneves ' post was suggesting we migrate everyone away from one of the format.

The argument I remember at the time of the SDL discussion was that, had SDL not been made the default immediately, things would have gone over a lot smoother. It also didn't help that the discussion was started by Andrei. As you can see from the diff, YAML will not be the default, and it will only take precedence of package.json, because no one should be using package.json nowadays.

Now, admittedly, I don't quite agree with the gravity of the fragmentation argument. It's more so that this whole discussion, which has been brought up time and time again, has cost me a lot of motivation and, apart from the objective aspects, it still bothers me a bit to now see someones favorite color of the shed just being sneaked in.

Given the amount of discussions we had on this, the feedback from the two BDFL, and the time that has passed, I hope that PR doesn't count as sneaking it in!

But regarding YAML itself, I really think the language design is flawed in many ways and not a technically good choice. The main argument is that it's frequently used for CI configuration files and in some other more or less related areas. But in terms of usability and simplicity, SDL is just superior for this use case.

I'm not going to argue here. But when I open dub.sdl in VSCode, I also don't get syntax highlighting (which include highlighting syntax errors). Same thing when I open it in Github. We don't have the capacity / people to build the tooling required to make everyone's life easier, and using YAML is a way to sidestep that problem. I'd rather spend time making DMD-as-a-library a reality than implementing SDL support for Github / GitLab / VSCode.

The fact that something is technically easy to implement alone is a very weak argument when it comes to designing a system - no matter whether Confiy just happens to be able to parse YAML or DMD conveniently has C bitfield semantics already implemented.

I strongly agree with this, and it became obvious to me, as soon as saw your message, I should have expanded on the rationale. Ease of implementation is not the reason we should do it. At most, ease of implementation helps build confidence that the feature works, and simplify code review, but it should never be a justification for the feature itself. The reason we should add support for YAML is because, at the moment, we have the choice between two formats:

1. JSON, which I think we all agree is terrible for human-written configuration;
2. SDL, which, while superior, has not gained traction, lacks support across the board;

YAML has its flaws, but is not a bad language, and has definitely gained a massive amount of traction. As discussed, everyone on Github is exposed to YAML. It's hard to expose how important the lack of tooling is, because every single example can be discussed, cherry picked, but it's really a death by a thousand cut. No out-of-the box support for editors / online code viewer, few parsers / emitter available (especially in other languages), little references, very poor support for LLM generation as the language is just now that common, etc...

Anyway, apart from all of this, this is a change with a strong long-term impact that IMO warrants a proper community discussion. And if we go ahead with this, there is also quite a bit of documentation to be amended (and maintained, in case of new recipe features).

I'm okay to revive the discussion, but I don't think the forum is particularly productive there. I know Walter and Atila are on board, or at least don't oppose it, and last time I mentioned it at the DLF meeting, I don't recall any objection.

[s-ludwig]

I don't want to get into the SDL/JSON discussion again, I'm tired of it, as the same arguments are repeated over and over again, everyone focusing on their own specific favorite point. As far as I see it, the real mistake happened earlier and was keeping the JSON support in the first release version of DUB, the transition plan to SDL had already been there for a while at that point and JSON was never really a good format for the purpose, it just happened to be convenient due to the existing implementation(s).

My stance at the time was against adding YAML support

I remember you pushing for YAML in multiple places and occasions, but I don't think it really matters in the end - I just think the now technically convenient possibility should receive the same scrutiny as the earlier package format topic, because a fundamental change like this has a strong long-term effect and we should carefully weigh the costs and benefits.

I hope that PR doesn't count as sneaking it in!

Yes, I should have put that in quotation marks. Back when the SDL/JSON discussions happened there simply was a lot of community activity around this, which is not really happening in this case (maybe because the forums were just more active in general, maybe because the topic just isn't as interesting anymore). It would have been unthinkable to just open a PR back then to deprecate JSON support or to add another format without causing huge backlash.

But when I open dub.sdl in VSCode, I also don't get syntax highlighting (which include highlighting syntax errors). Same thing when I open it in Github.

I actually implemented a syntax highlighting package for Sublime Text 10 years ago and I don't think it took me more than an evening and it should be easily adaptable to VSCode: https://github.com/s-ludwig/sublime-sdlang

GitHub appears to use third-party syntax highlighting modules, so that might then also be solvable with a simple support ticket: https://github.com/github-linguist/linguist/blob/main/vendor/README.md

SDL, which, while superior, has not gained traction, lacks support across the board;

Despite JSON being the default when creating new packages, SDL makes up about one third of the packages on code.dlang.org. My question would be though, why is anything other than syntax highlighting support even relevant for our use case more in-depth SDL reference, parsers in more foreign languages)?

Just for fun, querying Gemma3 (12b) to write a package recipe with some platform specific parts, it did hallucinate in the dependency and platform specific parts, but it nailed the SDL syntax. JSON was very similar in that regard. This might differ for larger models, of course.

I'm okay to revive the discussion, but I don't think the forum is particularly productive there.

Definitely, the forum is awful for anything that offers the opportunity for bike-shedding. However, I think the discussion should follow a similar structure and should not merely rely on a more ephemeral medium like a video conference or IRC/Slack.

The main points of concern that I see are long term vision (e.g. keep all three, narrow down on one/two?) and playing through different scenarios with an extended package format syntax (e.g. DEP3/DEP4). Also, do you have a complex example of how a complex YAML recipe would look like with this PR?

[Herringway]

I see no problems here that didn't also exist for the inclusion of SDL support, and since those seem to have been ignored with no consequence, it seems safe to ignore them once again. I would definitely appreciate a common format that supports commenting.

[s-ludwig]

@Herringway: SDL support has been there from the beginning (before DUB was first released as such), the discussion broke out when SDL was to be made the default format. Apart from that, as already said, the plan was to make it the only format that is human authored, not an additional one. Lastly, it's a fallacy to try to justify a supposed mistake with an analogous earlier supposed mistake.

[Herringway]

@Herringway: SDL support has been there from the beginning (before DUB was first released as such), the discussion broke out when SDL was to be made the default format. Apart from that, as already said, the plan was to make it the only format that is human authored, not an additional one. Lastly, it's a fallacy to try to justify a supposed mistake with an analogous earlier supposed mistake.

My earliest packages (https://github.com/Herringway/natcmp, for example) predate the initial work to support dub.sdl packages in 4393dad533e2ad8faa2924677b81c86524a10db3. I was around for this.

Precedence is not a fallacy. It isn't necessarily proof, either, but I am seeing no reason to disagree with it at this time.

[s-ludwig]

My earliest packages (https://github.com/Herringway/natcmp, for example) predate the initial work to support dub.sdl packages in https://github.com/dlang/dub/commit/4393dad533e2ad8faa2924677b81c86524a10db3. I was around for this.

This was way before it became the official package manager and its first official release in 2016. Anyway, the plan at that time was to replace JSON, being a permanent alternative is what came out of that infamous discussion.

Precedence is not a fallacy. It isn't necessarily proof, either, but I am seeing no reason to disagree with it at this time.

So if you shoot yourself in your left foot, it's okay to also shoot in the right one, because of precedence?

[Herringway]

My earliest packages (https://github.com/Herringway/natcmp, for example) predate the initial work to support dub.sdl packages in 4393dad. I was around for this.

This was way before it became the official package manager and its first official release in 2016. Anyway, the plan at that time was to replace JSON, being a permanent alternative is what came out of that infamous discussion.

Precedence is not a fallacy. It isn't necessarily proof, either, but I am seeing no reason to disagree with it at this time.

So if you shoot yourself in your left foot, it's okay to also shoot in the right one, because of precedence?

False equivalence is a fallacy. Shooting yourself in the foot has obvious negative consequences. Supporting a new format does not.

[s-ludwig]

They may not be as obvious, but there are all kinds of potentially negative impacts that need to be taken into account (maintenance and documentation costs, system complexity and implementation size, user confusion and communication overhead, possibly making it more difficult to make format changes or adding constraints on how format extensions can be represented are a few that come to mind). The fact that removing an existing format has a very high cost, means that those costs and the benefits should be weighted very carefully against each other.

[Herringway]

They may not be as obvious, but there are all kinds of potentially negative impacts that need to be taken into account (maintenance and documentation costs, system complexity and implementation size, user confusion and communication overhead, possibly making it more difficult to make format changes or adding constraints on how format extensions can be represented are a few that come to mind). The fact that removing an existing format has a very high cost, means that those costs and the benefits should be weighted very carefully against each other.

It's been over ten years since SDL support was added. We have ten years of data to measure these "potential" impacts with.

[Geod24]

I actually implemented a syntax highlighting package for Sublime Text 10 years ago and I don't think it took me more than an evening and it should be easily adaptable to VSCode: https://github.com/s-ludwig/sublime-sdlang

GitHub appears to use third-party syntax highlighting modules, so that might then also be solvable with a simple support ticket: https://github.com/github-linguist/linguist/blob/main/vendor/README.md

Note that the statement after the one you quote answers this: Why don't we have out of the box support for SDL in VSCode and Github ? Because:

We don't have the capacity / people to build the tooling required to make everyone's life easier, and using YAML is a way to sidestep that problem. I'd rather spend time making DMD-as-a-library a reality than implementing SDL support for Github / GitLab / VSCode.

I know about linguist, and when things broke for syntax highlighting on Github the issue was handled quickly. But it's been 10 years and Github still doesn't support SDL syntax highlighting despite the ease to do so, as you pointed out. Perhaps the problem is not with the difficulty of the task, then.

Despite JSON being the default when creating new packages, SDL makes up about one third of the packages on code.dlang.org. My question would be though, why is anything other than syntax highlighting support even relevant for our use case more in-depth SDL reference, parsers in more foreign languages)?

I'm not sure I follow the question. As pointed out earlier, this is death by a thousands (paper) cuts: unfamiliarity, lack of tooling, lack of libraries.

Definitely, the forum is awful for anything that offers the opportunity for bike-shedding. However, I think the discussion should follow a similar structure and should not merely rely on a more ephemeral medium like a video conference or IRC/Slack.

I think, before said discussion happens, we should define the criterias for rejecting or approving it. We've always followed a BDFL approach, so I think if both Walter and Atila are in agreement, we should go ahead with it, unless you have another suggestion. The discussion can happen in the next foundation / industry meeting if you'd like and are available. If there are requirements to this happening, such as deprecating JSON, they can also be voiced and agreed upon there. What do you think ?

The main points of concern that I see are long term vision (e.g. keep all three, narrow down on one/two?) and playing through different scenarios with an extended package format syntax (e.g. DEP3/DEP4). Also, do you have a complex example of how a complex YAML recipe would look like with this PR?

I think we should add support for YAML and gradually deprecate JSON and SDL going forward, first by removing it from dub init, then by warning on building projects using them, then finally by erroring on projects that use it. When it comes to packages being used from the registry, I would like to keep supporting both package recipe for an extended period of time (we can discuss in terms of releases, or years), as that would be the most disruptive part.

When it comes to extend the package format, that was not yet in scope. I would not like to bundle too many changes at once, but perhaps it's an area of discussion as well.

[s-ludwig]

Note that the statement after the one you quote answers this: Why don't we have out of the box support for SDL in VSCode and Github ? Because:

We don't have the capacity / people to build the tooling required to make everyone's life easier, and using YAML is a way to sidestep that problem. I'd rather spend time making DMD-as-a-library a reality than implementing SDL support for Github / GitLab / VSCode.

I'm stating that writing a syntax highlighting file is minimal effort and already done and you reply with "We don't have the capacity / people to build the tooling required"? I'm pretty sure that doing the YAML addition/transition requires a lot more capacity and the impact of missing GitHub/GitLab support doesn't seem to be particularly high.

I know about linguist, and when things broke for syntax highlighting on Github the issue was handled quickly. But it's been 10 years and Github still doesn't support SDL syntax highlighting despite the ease to do so, as you pointed out. Perhaps the problem is not with the difficulty of the task, then.

Perhaps the issue is simply that nobody has made a request to add support based on an existing repository? There are lots if obscure formats in the list and I don't expect them to just constantly search the web for new formats to support.

My question would be though, why is anything other than syntax highlighting support even relevant for our use case more in-depth SDL reference, parsers in more foreign languages)?

I'm not sure I follow the question. As pointed out earlier, this is death by a thousands (paper) cuts: unfamiliarity, lack of tooling, lack of libraries.

I'm asking why you think that lack of libraries is relevant to DUB and I'm asking why you think that "little references" are relevant. I'm asking that in a concrete way, as I cannot see where you see the "thousand cuts" theory playing out in practice - to me this is just an empty phrase without at least some kind of evidence.

We've always followed a BDFL approach, so I think if both Walter and Atila are in agreement, we should go ahead with it, unless you have another suggestion.

I don't care how the final choice is made, whether this is a decision from the top, a community vote or Herringway who decides this. What I do care about is that this is an informed decision with a well founded long-term vision. Discussing this in the next meeting makes sense, although I don't think that that's a particularly good medium for working out a vision (the forum format IMO not bad for this, it should just be a smaller group of people involved).

I think we should add support for YAML and gradually deprecate JSON and SDL going forward.

This honestly sounds like an extremely bad trade-off in terms of ecosystem health, even if YAML would ultimately be deemed superior. A change that breaks all packages in existence for zero benefit to those packages, except maybe in cases where the package author is just happy to switch to YAML. It's exactly the kind of change that you'd want to avoid. The only way I can see this being a net positive would be if we assume that the community grows heavily in the coming years, so that eventually the advantages of the new only format outweigh the damage done by the removal.

[LightBender]

Some thoughts regarding this discussion:

While GitHub/Gitlab do not support SDL, VSCode has supported SDL OOB for a while now: There are two extensions if you want more than that. I suspect getting SDL into Github/Gitlab is more of a "Time and will power" problem than anything else. I very much doubt that they've specifically rejected it. IMO, GitHub and Gitlab not supporting SDL is more of an argument for doing the work to get it supported, rather than an argument against retiring/replacing it.

Personally, I find working with YAML to be a dreadful experience full of weird gotchas and my default in DUB is SDL. I also use SDL in the tools that I write in D. I would like to echo Atila's comments from the linked Forum post that adding a third language to DUB would only make things worse.

If I were to vote for anything, I would vote for deprecating JSON and removing it from the new project menu. We can still read the JSON format for backwards compatibility, but all new projects are SDL only. This would be the most beneficial course of action from a usability standpoint, and we could remove the confusing "Two document pages for the same thing" problem.

[Herringway]

I think we should add support for YAML and gradually deprecate JSON and SDL going forward

Agreed. JSON is deficient (lack of proper comments are a pretty common complaint) and SDL is just too obscure. However, I don't think it's reasonable to remove them, since there are a number of packages will never be updated... Perhaps if enough of the packages also fail to compile?

[Herringway]

I suspect getting SDL into Github/Gitlab is more of a "Time and will power" problem than anything else. I very much doubt that they've specifically rejected it. IMO, GitHub and Gitlab not supporting SDL is more of an argument for doing the work to get it supported, rather than an argument against retiring/replacing it.

You may be waiting a very long time for someone to volunteer for this.

[Herringway]

I would also like to point out that since dub has been using a YAML parser to parse dub.json files without JSON validation for some time now, it's quite possible that some misnamed dub.yaml files already exist in the wild.

[LightBender]

So what I gather from this thread is that the only compelling reason to make this highly disruptive change is that GitHub/Gitlab support YAML syntax highlighting?

I'm struggling to see why that merits such a disruptive change.

[schveiguy]

I vehemently disagree with yaml support (my position has not changed). It's a terrible format, and adding more format support isn't going to fix any of dub's real problems. It will just add to the problems.

Will bring this page up again:

https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-from-hell

My recommendation has been if we want to support a better format to go with json5. But better still, we should focus on the existing problems in dub before adding more formats. Maybe start with making it so dub can edit its known file formats without destroying the formatting? #2575

[Geod24]

So what I gather from this thread is that the only compelling reason to make this highly disruptive change is that GitHub/Gitlab support YAML syntax highlighting?

I'm struggling to see why that merits such a disruptive change.

No, the main reason is familiarity. YAML is by far more common, which leads it to being much better supported everywhere, of which syntax highlighting is an example. What is SDL's main use case ? Dub recipe files. Where else is it used prominently ? YAML is used everywhere, and aspiring D users will be able to pick it up much faster than SDL. It will also make their life easier because of the breadth of tooling available.

Will bring this page up again:

https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-from-hell

All of those problems are fixed by quoting strings. It's not rocket science.

[Geod24]

I'm stating that writing a syntax highlighting file is minimal effort and already done and you reply with "We don't have the capacity / people to build the tooling required"? I'm pretty sure that doing the YAML addition/transition requires a lot more capacity and the impact of missing GitHub/GitLab support doesn't seem to be particularly high.

"Building the tooling" is not solely about writing code. It's also integrating it and maintaining it. We're all experienced programmers here and we all experienced how much maintainance actually cost compared to the initial development time.

I'm asking why you think that lack of libraries is relevant to DUB and I'm asking why you think that "little references" are relevant. I'm asking that in a concrete way, as I cannot see where you see the "thousand cuts" theory playing out in practice - to me this is just an empty phrase without at least some kind of evidence.

D's ecosystem weakest point is tooling, I hope we can agree with this. To me this is a consequence of being spread to thin, something exacerbed by our NIH tendencies. Today if I want to build tooling for D that somehow uses the recipe file, not only do I have to build the tooling, but most likely I have to select among a short set of SDL parsing libraries, or build one myself. But then I also have to make sure I provide good error handling, and god forbid if we ever want to update the SDL format.

I don't care how the final choice is made, whether this is a decision from the top, a community vote or Herringway who decides this. What I do care about is that this is an informed decision with a well founded long-term vision. Discussing this in the next meeting makes sense, although I don't think that that's a particularly good medium for working out a vision (the forum format IMO not bad for this, it should just be a smaller group of people involved).

Appreciated.

This honestly sounds like an extremely bad trade-off in terms of ecosystem health, even if YAML would ultimately be deemed superior. A change that breaks all packages in existence for zero benefit to those packages, except maybe in cases where the package author is just happy to switch to YAML. It's exactly the kind of change that you'd want to avoid. The only way I can see this being a net positive would be if we assume that the community grows heavily in the coming years, so that eventually the advantages of the new only format outweigh the damage done by the removal.

If it was up to me, I would let SDL / JSON fade for the foreseable future because I'm sure YAML is going to be widely adopted. But the opinion expressed by one of the BDFL already shows that's not going to be a workable path. The timelines for doing so can be quite long though, which would have a similar impact. And as already mentioned, we can put that restriction on projects, which by definition would only affect actively maintained projects.

[schveiguy]

What is SDL's main use case ? Dub recipe files. Where else is it used prominently ?

SDL is never going away. Anything we add here is adding, SDL will always be there, because nobody is going to go back through all released revisions of all projects and change dub.sdl to dub.yaml.

We will continue to have the problems of SDL, and add all the problems of YAML.

This goes for any new format we add. We should only add a format if it's worth adding on its own.

If you want a prominently used format, use JSON. There is plenty of tooling for JSON.

All of those problems are fixed by quoting strings.

So we will have a format where it's easy to put in bizarre corner cases, and our response to users who have those issues is going to be that it's YAML, so you can write bad formats, nothing we can do? This reeks of "you're holding it wrong" scolding.

The beauty of JSON is that it is never ambiguous. It's definitive, and there are no "options" when it comes to how to specify things.

last time I mentioned it at the DLF meeting, I don't recall any objection.

I missed this before. I have objected every time I'm at the DLF meeting. I shouldn't have to attend them all to bring this up every time.

[Geod24]

The beauty of JSON is that it is never ambiguous. It's definitive, and there are no "options" when it comes to how to specify things.

There are options, one of them you've been recommending (JSON5). There's also JSONC (https://en.wikipedia.org/wiki/JSON#Supersets), and obviously YAML.

SDL is never going away. Anything we add here is adding, SDL will always be there, because nobody is going to go back through all released revisions of all projects and change dub.sdl to dub.yaml.

We will continue to have the problems of SDL, and add all the problems of YAML.

We need to continue supporting reading the current SDL format. That's it. We can get rid of SDL formatting code (support as target in dub convert), we don't need to support it in dub add (the aforementioned bug), and we don't need to support it going forward (can get rid of documentation, don't have to think about it if we upgrade the recipe format...).

Because YAML is a superset of JSON, we get it for (almost) free. SDL has a cost. Going with JSON5 would be less disturbing than YAML, and the upside is IMO pretty limited.

[schveiguy]

we don't need to support it going forward (can get rid of documentation, don't have to think about it if we upgrade the recipe format...).

So when someone looks at a package, and reads the dub.sdl, what are they going to do? How do they look up how to edit that file, or what things mean?

The documentation is already there, why would we ever get rid of it?

I don't see the point of removing SDL, even if we don't want to encourage it. We literally cannot get rid of it, because of existing packages. And if we have to support it, we should keep documenting the support.

[s-ludwig]

"Building the tooling" is not solely about writing code. It's also integrating it and maintaining it. We're all experienced programmers here and we all experienced how much maintainance actually cost compared to the initial development time.

There can, however, be extreme differences in the ratio here. In the concrete example of the syntax highlighting module, maintenance cost has been more or less zero.

I'm asking why you think that lack of libraries is relevant to DUB and I'm asking why you think that "little references" are relevant. I'm asking that in a concrete way, as I cannot see where you see the "thousand cuts" theory playing out in practice - to me this is just an empty phrase without at least some kind of evidence.

D's ecosystem weakest point is tooling, I hope we can agree with this. To me this is a consequence of being spread to thin, something exacerbed by our NIH tendencies. Today if I want to build tooling for D that somehow uses the recipe file, not only do I have to build the tooling, but most likely I have to select among a short set of SDL parsing libraries, or build one myself. But then I also have to make sure I provide good error handling, and god forbid if we ever want to update the SDL format.

If you want to make a tool that interacts with the recipe file, you should use dub as a library and not reinvent the whole package parsing and higher level logic - that just begs for problems as the package format evolves and multiplies maintenance costs. But the more important point is, having a third format that needs to be supported doesn't make for a good argument in terms of facilitating easier writing of tooling. Even if everything but YAML would be dropped 10 years from now, you would still have made it harder to write tooling until then. And just not supporting the existing formats will not improve the tooling landscape, but will just make it appear more broken.

[CyberShadow]

I vehemently disagree with yaml support (my position has not changed). It's a terrible format, and adding more format support isn't going to fix any of dub's real problems. It will just add to the problems.

I agree that YAML has flaws. I also agree that better formats exist. However, we need to separate our personal feelings, looking for the perfect solution, and making a decision which benefits the project. It is quite clear that YAML won this particular format war - it's used by GitHub, GitLab, all the other big CI providers, Kubernetes, Ansible, Docker... we could revisit this in a few years, but fairly sure the list will have only grown by then.

So when someone looks at a package, and reads the dub.sdl, what are they going to do? How do they look up how to edit that file, or what things mean?

By the time SDL support in Dub is deprecated and then removed, they would look at the documentation of the Dub version from the time the .sdl file was written, just like how you would need to consult an old version of the Phobos documentation to see what a deprecated-then-removed function did.

The documentation is already there, why would we ever get rid of it?

Even if we delete it from a new version of Dub, old versions are still available for download. Maybe we could put old Dub documentation on https://docarchives.dlang.io/ if we need to.

We literally cannot get rid of it, because of existing packages.

Sure we can. We have a deprecation policy for things like language features and Phobos functions, we can do the same thing here. What's the point of supporting a package definition format from 20 years ago if none of the programs from then will build with the current compiler anyway?

There can, however, be extreme differences in the ratio here. In the concrete example of the syntax highlighting module, maintenance cost has been more or less zero.

Yes, I haven't needed to update the Emacs package for SDLang since I wrote it 8 years ago. However...

1. I suspect that part of the challenge is going to be to convince providers (e.g. GitHub and GitLab) that SDLang is sufficiently notable that it is worthy of inclusion, even if someone does all the work.
2. Providers can change syntax highlighting libraries at any time - I recall GitHub and GitLab has done so at least once, thus throwing away all the work done to support the less popular formats so far.

[schveiguy]

My objections to YAML aren't personal. It's objectively a bad format. Yes, I understand that it has "won" when it comes to people being forced to use it. However, the plan here shouldn't be to force people to use YAML. This would be a bad move IMO.

The existence of SDL does not discourage people from using D. Removing SDL would definitely alienate existing users. Forcing YAML on people would definitely alienate users. Adding YAML as an option will not gain users (nobody is going to use D now because dub does YAML). The discussion here is literally only about us here, do we want YAML or not? All of us will continue to use D whether YAML is there or not. There is no checkboxes on anyones list that this helps with.

It costs almost nothing to support SDL. Why is SDL such a bad format that we should remove it? Nobody has explained this. The same goes for JSON. Note that neither SDL nor JSON formats have changed. Not including new dub features in JSON or SDL would be mostly spiteful. You would probably have to reject PRs that fix any issues.

I have multiple projects that I inherited (and some that I started) that use SDL as the dub format. I don't use it for new projects, because JSON is easier to use for me. But I certainly don't want to switch to JSON on those or YAML, SDL is fine to use for maintenance.

If YAML gets added, I will be sad, but move on. If SDL or JSON gets deprecated, I will be mad. Deprecating is a bad move here that gains us nothing, and just adds friction to using D.

[CyberShadow]

Forcing YAML on people would definitely alienate users.

Let's please see these people who have somehow avoided GitHub, GitLab, all the other big CI providers, Kubernetes, Ansible, Docker, and plan on avoiding all future software that will use what is probably the most common configuration file format used today...

Sorry, it's difficult for me to empathize with this. I can't see this as anything but a radical personal opinion.

It costs almost nothing to support SDL. Why is SDL such a bad format that we should remove it? Nobody has explained this.

There is a rather long discussion above that explains this. To bring it into context: the merits of the file format itself are almost entirely irrelevant, the popularity is what actually matters.

If YAML gets added, I will be sad, but move on. If SDL or JSON gets deprecated, I will be mad. Deprecating is a bad move here that gains us nothing, and just adds friction to using D.

Understandable, but please also take into account that this is one Steven Schveighoffer versus many people who either don't care or would be happy to just use a format they're already familiar with, that their text editor can syntax-highlight / auto-format / auto-syntax-check etc.

[Geod24]

If YAML gets added, I will be sad, but move on. If SDL or JSON gets deprecated, I will be mad. Deprecating is a bad move here that gains us nothing, and just adds friction to using D.

Note that the time horizon for deprecation is long. Honestly long. The steps would be:

1. Add support for it, and ensure we have good documentation / examples;
2. Wait a X releases because you should never introduce a deprecation at the same time as you introduce the alternative;
3. During (2), make YAML the default;
4. Start to issue deprecation warnings for projects using dub.json / dub.sdl;
5. After Y releases, start to issue deprecation warnings for packages using dub.json / dub.sdl;
6. After Z releases, remove support for dub.json / dub.sdl;

If X = 4, Y = 10, Z = 10, you're looking at 6 years before we issue an error because your dependency uses dub.sdl. We could make it longer, or even not do (6), if the cost of converting old packages will be kept low-ish. Sometimes between (4) and (5), or maybe coupled with (5), we would also stop recognizing dub.sdl / dub.json. But that would only affect actively developed packages which are likely to transition easily.