DevOps

Nix comes closest. The biggest issue with Nix is that it does not admit a clean simple LL/LR grammar, but I think it admits a PEG, which is fine for practical work. The ecosystem could use more independent implementations and tooling, but I haven't found any deficiencies with the language that would make me prefer e.g. Guile Scheme's larger standard library and richer types.

HCL is another option. It's not awful, but it's verbose when trying to do anything higher-order, and it wasn't intended to directly represent lambda-style subroutines. Guile Scheme, as used in Guix, is clearly capable, but requires embedding a fairly large standard library or having to NIH common routines; similar problems plague Lua or Python.

I think that your question has an interesting framing. My first answer was to mention jq and the relational pipes toolkit, but those are still run from a top-level shell. For example, I suppose that there's two things that you can do with JSON: you can manipulate it and I would use jq for that, or you can load it as a datastructure into a (build) action and I would use Nix for that. jq is close to ideal for its particular formalization, but relational pipes are still evolving and I think that there are further simplifications that could be made.

There isn’t a universal language to fill all these needs. DevOps covers a ton of areas so you want several tools in your toolbelt instead of just one (granted if you’re an incredibly talented greybeard bash can honestly do everything and do it fairly fast; I’ve never taken this analysis too seriously).

• bash is something you need for every day tasks and quick tooling to improve everything else in this list
• POSIX tooling is something you should know well to improve everything else on this list (eg awk is something you’ll use a lot in bash)
• Python with or without types will handle more advanced scripting in a cleaner and more maintainable way for your team (explaining bashisms slows things down. Does your junior know what : is or why we currently prefer $() over backticks?)
• Go is something you want to use for containerization and scaling servers
• Rust is something you want to use for process work and embeddables

Each of these languages meets some of the requirements on your list but not all. That’s because nothing is a silver bullet. Depending on what you’re doing, you might even want to introduce, say, Java or C# in your pipelines if that’s what you’re maintaining. DevOps should support not alienate and overall it needs to be flexible.

I don't have an ideal, but a go to Usually python scripts...

Declarative systems are so much better to work in the domain, why would you force yourself into "pipeline logic"?

The advantage of yaml is idempotency, so you really should be using generators to produce outputs that you then check in. This gives you a state repo, which when tools feed from to produce a final state, you always have the same result.

Check out CDK for an example.

I prefer pyinfra over ansible.

I like https://github.com/bazelbuild/starlark as a concept, but it is sadly quite unpopular.

Most of them are bash scripts as they usually don't need any specific tools installed and run out of the box. When it gets more complicated (especially if there are lots of asynchronous and parallel tasks) I go to Node.js / Typescript. This is maybe a little bit of an oddball, but there are plenty of tools to create CLI binaries that can run independently and working asynchronously is really easy. I like the language a lot more than I like Python – I haven't tried Go yet.

I personally like golang a lot for small-ish scripting and programming. It has strong typing and few foot guns (e.g. bash with the weird way arrays are handled, implicit splitting of array items etc). Downside is it needs to be compiled up front and is basically a black box from the CI configuration's POV (you can inspect the source of a circleci orb or GitHub action for example).

I use terraform, helm, a bit of ansible and gitlab/forgejo (for cicd). The minimum amount of scripting I do is shell, I don't have a need for more...

Back in the days of manual deployments and semi-automated configurations, I did use bash.

What is your need for complex scripting/programming?

Most things I'd say is bash, easy just to write a simple task in it. If you have to use some more logic I use python. But python is far from great for it. It is easy for me to run a venv and test the stuff. But "this script should do a cron thing", "this is an filter for ansible", "this performs a job here or there", managing a uniform consistent env for all of it is a mess. Bjt yeah, if it is just pipeline logic, ok that env can be managed.